[AST] Add OriginalDC argument to ExternalASTSource::FindExternalVisibleDeclsByName (#123152)

Part for relanding https://github.com/llvm/llvm-project/pull/122887.

I split this to test where the pe

[AST] Add OriginalDC argument to ExternalASTSource::FindExternalVisibleDeclsByName (#123152)

Part for relanding https://github.com/llvm/llvm-project/pull/122887.

I split this to test where the performance regession comes from if
modules are not used.

show more ...

Revert "[C++20] [Modules] Support module level lookup (#122887)"

This reverts commit 7201cae106260aeb3e9bbbb7d5291ff30f05076a.

[C++20] [Modules] Support module level lookup (#122887)

Close https://github.com/llvm/llvm-project/issues/90154

This patch is also an optimization to the lookup process to utilize the
informatio

[C++20] [Modules] Support module level lookup (#122887)

Close https://github.com/llvm/llvm-project/issues/90154

This patch is also an optimization to the lookup process to utilize the
information provided by `export` keyword.

Previously, in the lookup process, the `export` keyword only takes part
in the check part, it doesn't get involved in the lookup process. That
said, previously, in a name lookup for 'name', we would load all of
declarations with the name 'name' and check if these declarations are
valid or not. It works well. But it is inefficient since it may load
declarations that may not be wanted.

Note that this patch actually did a trick in the lookup process instead
of bring module information to DeclarationName or considering module
information when deciding if two declarations are the same. So it may
not be a surprise to me if there are missing cases. But it is not a
regression. It should be already the case. Issue reports are welcomed.

In this patch, I tried to split the big lookup table into a lookup table
as before and a module local lookup table, which takes a combination of
the ID of the DeclContext and hash value of the primary module name as
the key. And refactored `DeclContext::lookup()` method to take the
module information. So that a lookup in a DeclContext won't load
declarations that are local to **other** modules.

And also I think it is already beneficial to split the big lookup table
since it may reduce the conflicts during lookups in the hash table.

BTW, this patch introduced a **regression** for a reachability rule in
C++20 but it was false-negative. See
'clang/test/CXX/module/module.interface/p7.cpp' for details.

This patch is not expected to introduce any other
regressions for non-c++20-modules users since the module local lookup
table should be empty for them.

---

On the API side, this patch unfortunately add a maybe-confusing argument
`Module *NamedModule` to
`ExternalASTSource::FindExternalVisibleDeclsByName()`. People may think
we can get the information from the first argument `const DeclContext
*DC`. But sadly there are declarations (e.g., namespace) can appear in
multiple different modules as a single declaration. So we have to add
additional information to indicate this.

show more ...

[Serialization] Support loading template specializations lazily (#119333)

Reland https://github.com/llvm/llvm-project/pull/83237

---

(Original comments)

Currently all the specializations of

[Serialization] Support loading template specializations lazily (#119333)

Reland https://github.com/llvm/llvm-project/pull/83237

---

(Original comments)

Currently all the specializations of a template (including
instantiation, specialization and partial specializations) will be
loaded at once if we want to instantiate another instance for the
template, or find instantiation for the template, or just want to
complete the redecl chain.

This means basically we need to load every specializations for the
template once the template declaration got loaded. This is bad since
when we load a specialization, we need to load all of its template
arguments. Then we have to deserialize a lot of unnecessary
declarations.

For example,

```
// M.cppm
export module M;
export template <class T>
class A {};

export class ShouldNotBeLoaded {};

export class Temp {
A<ShouldNotBeLoaded> AS;
};

// use.cpp
import M;
A<int> a;
```

We have a specialization ` A<ShouldNotBeLoaded>` in `M.cppm` and we
instantiate the template `A` in `use.cpp`. Then we will deserialize
`ShouldNotBeLoaded` surprisingly when compiling `use.cpp`. And this
patch tries to avoid that.

Given that the templates are heavily used in C++, this is a pain point
for the performance.

This patch adds MultiOnDiskHashTable for specializations in the
ASTReader. Then we will only deserialize the specializations with the
same template arguments. We made that by using ODRHash for the template
arguments as the key of the hash table.

To review this patch, I think `ASTReaderDecl::AddLazySpecializations`
may be a good entry point.

show more ...

Revert "[Serialization] Support load lazy specialization lazily"

This reverts commit b5bd19211118c6d43bc525a4e3fb65d2c750d61e.
It brokes multiple llvm bots including clang-x64-windows-msvc

[Serialization] Support load lazy specialization lazily

Currently all the specializations of a template (including
instantiation, specialization and partial specializations) will be
loaded at once

[Serialization] Support load lazy specialization lazily

Currently all the specializations of a template (including
instantiation, specialization and partial specializations) will be
loaded at once if we want to instantiate another instance for the
template, or find instantiation for the template, or just want to
complete the redecl chain.

This means basically we need to load every specializations for the
template once the template declaration got loaded. This is bad since
when we load a specialization, we need to load all of its template
arguments. Then we have to deserialize a lot of unnecessary
declarations.

For example,

```
// M.cppm
export module M;
export template <class T>
class A {};

export class ShouldNotBeLoaded {};

export class Temp {
A<ShouldNotBeLoaded> AS;
};

// use.cpp
import M;
A<int> a;
```

We should a specialization ` A<ShouldNotBeLoaded>` in `M.cppm` and we
instantiate the template `A` in `use.cpp`. Then we will deserialize
`ShouldNotBeLoaded` surprisingly when compiling `use.cpp`. And this
patch tries to avoid that.

Given that the templates are heavily used in C++, this is a pain point
for the performance.

This patch adds MultiOnDiskHashTable for specializations in the
ASTReader. Then we will only deserialize the specializations with the
same template arguments. We made that by using ODRHash for the template
arguments as the key of the hash table.

To review this patch, I think `ASTReaderDecl::AddLazySpecializations`
may be a good entry point.

The patch was reviewed in
https://github.com/llvm/llvm-project/pull/83237 but that PR is a stacked
PR. But I feel the intention of the stacked PRs get lost during the
review process. So I feel it is better to merge the commits into a
single commit instead of merging them in the PR page. It is better for
us to cherry-pick and revert.

show more ...

[C++20] [Modules] Merge lambdas in source to imported lambdas (#106483)

Close https://github.com/llvm/llvm-project/issues/102721

Generally, the type of merged decls will be reused in ASTContext.

[C++20] [Modules] Merge lambdas in source to imported lambdas (#106483)

Close https://github.com/llvm/llvm-project/issues/102721

Generally, the type of merged decls will be reused in ASTContext. But
for lambda, in the import and then include case, we can't decide its
previous decl in the imported modules so that we can't assign the
previous decl before creating the type for it. Since we can't decide its
numbering before creating it. So we have to assign the previous decl and
the canonical type for it after creating it, which is unusual and
slightly hack.

show more ...

[NFC] [Serialization] Avoid using DeclID directly as much as possible

This patch tries to remove all the direct use of DeclID except the real
low level reading and writing. All the use of DeclID is

[NFC] [Serialization] Avoid using DeclID directly as much as possible

This patch tries to remove all the direct use of DeclID except the real
low level reading and writing. All the use of DeclID is converted to
the use of LocalDeclID or GlobalDeclID. This is helpful to increase the
readability and type safety.

show more ...

Revert "[NFC] [Serialization] Avoid using DeclID directly as much as possible"

This reverts commit 42070a5c092ed420bf92ebf38229c594885e94c7.

I forgot to touch lldb.

[NFC] [Serialization] Avoid using DeclID directly as much as possible

This patch tries to remove all the direct use of DeclID except the real
low level reading and writing. All the use of DeclID is

[NFC] [Serialization] Avoid using DeclID directly as much as possible

This patch tries to remove all the direct use of DeclID except the real
low level reading and writing. All the use of DeclID is converted to
the use of LocalDeclID or GlobalDeclID. This is helpful to increase the
readability and type safety.

show more ...

[NFC] Move DeclID from serialization/ASTBitCodes.h to AST/DeclID.h (#89873)

Previously, the DeclID is defined in serialization/ASTBitCodes.h under
clang::serialization namespace. However, actually

[NFC] Move DeclID from serialization/ASTBitCodes.h to AST/DeclID.h (#89873)

Previously, the DeclID is defined in serialization/ASTBitCodes.h under
clang::serialization namespace. However, actually the DeclID is not
purely used in serialization part. The DeclID is already widely used in
AST and all around the clang project via classes like `LazyPtrDecl` or
calling `ExternalASTSource::getExernalDecl()`. All such uses are via the
raw underlying type of `DeclID` as `uint32_t`. This is not pretty good.

This patch moves the DeclID class family to a new header `AST/DeclID.h`
so that the whole project can use the wrapped class `DeclID`,
`GlobalDeclID` and `LocalDeclID` instead of the raw underlying type.
This can improve the readability and the type safety.

show more ...

[NFC] [Serialization] Use semantical type DeclID instead of raw type 'uint32_t'

This patch tries to use DeclID in the code bases to avoid use the raw
type 'uint32_t'. It is problematic to use the ra

[NFC] [Serialization] Use semantical type DeclID instead of raw type 'uint32_t'

This patch tries to use DeclID in the code bases to avoid use the raw
type 'uint32_t'. It is problematic to use the raw type 'uint32_t' if we
want to change the type of DeclID some day.

show more ...

Revert "[Clang] Eagerly instantiate used constexpr function upon definition. (#73463)"

This reverts commit 030047c432cac133738be68fa0974f70e69dd58d.

Breaks Qt and is inconsistent with GCC.

See the

Revert "[Clang] Eagerly instantiate used constexpr function upon definition. (#73463)"

This reverts commit 030047c432cac133738be68fa0974f70e69dd58d.

Breaks Qt and is inconsistent with GCC.

See the following issue for details:

Fixes #74069

show more ...

[clang] Remove extra ';' in MultiplexExternalSemaSource.cpp (NFC)

/llvm-project/clang/lib/Sema/MultiplexExternalSemaSource.cpp:317:2:
error: extra ';' outside of a function is incompatible with C++9

[clang] Remove extra ';' in MultiplexExternalSemaSource.cpp (NFC)

/llvm-project/clang/lib/Sema/MultiplexExternalSemaSource.cpp:317:2:
error: extra ';' outside of a function is incompatible with C++98 [-Werror,-Wc++98-compat-extra-semi]
};
^
1 error generated.

show more ...

[Clang] Eagerly instantiate used constexpr function upon definition. (#73463)

Despite CWG2497 not being resolved, it is reasonable to expect the
following code to compile (and which is supported by

[Clang] Eagerly instantiate used constexpr function upon definition. (#73463)

Despite CWG2497 not being resolved, it is reasonable to expect the
following code to compile (and which is supported by other compilers)

```cpp
template<typename T> constexpr T f();
constexpr int g() { return f<int>(); } // #1
template<typename T> constexpr T f() { return 123; }
int k[g()];
// #2
```

To that end, we eagerly instantiate all referenced specializations of
constexpr functions when they are defined.

We maintain a map of (pattern, [instantiations]) independent of
`PendingInstantiations` to avoid having to iterate that list after each
function definition.

We should apply the same logic to constexpr variables, but I wanted to
keep the PR small.

Fixes #73232

show more ...

PR60985: Fix merging of lambda closure types across modules.

Previously, distinct lambdas would get merged, and multiple definitions
of the same lambda would not get merged, because we attempted to

PR60985: Fix merging of lambda closure types across modules.

Previously, distinct lambdas would get merged, and multiple definitions
of the same lambda would not get merged, because we attempted to
identify lambdas by their ordinal position within their lexical
DeclContext. This failed for lambdas within namespace-scope variables
and within variable templates, where the lexical position in the context
containing the variable didn't uniquely identify the lambda.

In this patch, we instead identify lambda closure types by index within
their context declaration, which does uniquely identify them in a way
that's consistent across modules.

This change causes a deserialization cycle between the type of a
variable with deduced type and a lambda appearing as the initializer of
the variable -- reading the variable's type requires reading and merging
the lambda, and reading the lambda requires reading and merging the
variable. This is addressed by deferring loading the deduced type of a
variable until after we finish recursive deserialization.

This also exposes a pre-existing subtle issue where loading a
variable declaration would trigger immediate loading of its initializer,
which could recursively refer back to properties of the variable. This
particularly causes problems if the initializer contains a
lambda-expression, but can be problematic in general. That is addressed
by switching to lazily loading the initializers of variables rather than
always loading them with the variable declaration. As well as fixing a
deserialization cycle, that should improve laziness of deserialization
in general.

LambdaDefinitionData had 63 spare bits in it, presumably caused by an
off-by-one-error in some previous change. This change claims 32 of those bits
as a counter for the lambda within its context. We could probably move the
numbering to separate storage, like we do for the device-side mangling number,
to optimize the likely-common case where all three numbers (host-side mangling
number, device-side mangling number, and index within the context declaration)
are zero, but that's not done in this change.

Fixes #60985.

Reviewed By: #clang-language-wg, aaron.ballman

Differential Revision: https://reviews.llvm.org/D145737

show more ...

[NFC] Make MultiplexExternalSemaSource own sources

This change refactors the MuiltiplexExternalSemaSource to take ownership
of the underlying sources. As a result it makes a larger cleanup of
extern

[NFC] Make MultiplexExternalSemaSource own sources

This change refactors the MuiltiplexExternalSemaSource to take ownership
of the underlying sources. As a result it makes a larger cleanup of
external source ownership in Sema and the ChainedIncludesSource.

Reviewed By: aaron.ballman, aprantl

Differential Revision: https://reviews.llvm.org/D133158

show more ...

[clang][AST] Improve AST Reader/Writer memory footprint

Reduce memory footprint of AST Reader/Writer:
1. Adjust internal data containers' element type.
2. Switch to set for deduplication of deferred

[clang][AST] Improve AST Reader/Writer memory footprint

Reduce memory footprint of AST Reader/Writer:
1. Adjust internal data containers' element type.
2. Switch to set for deduplication of deferred diags.

Differential Revision: https://reviews.llvm.org/D101793

show more ...

Make iteration over the DeclContext::lookup_result safe.

The idiom:
```
DeclContext::lookup_result R = DeclContext::lookup(Name);
for (auto *D : R) {...}
```

is not safe when in the loop body we tr

Make iteration over the DeclContext::lookup_result safe.

The idiom:
```
DeclContext::lookup_result R = DeclContext::lookup(Name);
for (auto *D : R) {...}
```

is not safe when in the loop body we trigger deserialization from an AST file.
The deserialization can insert new declarations in the StoredDeclsList whose
underlying type is a vector. When the vector decides to reallocate its storage
the pointer we hold becomes invalid.

This patch replaces a SmallVector with an singly-linked list. The current
approach stores a SmallVector<NamedDecl*, 4> which is around 8 pointers.
The linked list is 3, 5, or 7. We do better in terms of memory usage for small
cases (and worse in terms of locality -- the linked list entries won't be near
each other, but will be near their corresponding declarations, and we were going
to fetch those memory pages anyway). For larger cases: the vector uses a
doubling strategy for reallocation, so will generally be between half-full and
full. Let's say it's 75% full on average, so there's N * 4/3 + 4 pointers' worth
of space allocated currently and will be 2N pointers with the linked list. So we
break even when there are N=6 entries and slightly lose in terms of memory usage
after that. We suspect that's still a win on average.

Thanks to @rsmith!

Differential revision: https://reviews.llvm.org/D91524

show more ...

Merge some of the PCH object support with modular codegen

I was trying to pick this up a bit when reviewing D48426 (& perhaps D69778) - in any case, looks like D48426 added a module level flag that

Merge some of the PCH object support with modular codegen

I was trying to pick this up a bit when reviewing D48426 (& perhaps D69778) - in any case, looks like D48426 added a module level flag that might not be needed.

The D48426 implementation worked by setting a module level flag, then code generating contents from the PCH a special case in ASTContext::DeclMustBeEmitted would be used to delay emitting the definition of these functions if they came from a Module with this flag.

This strategy is similar to the one initially implemented for modular codegen that was removed in D29901 in favor of the modular decls list and a bit on each decl to specify whether it's homed to a module.

One major difference between PCH object support and modular code generation, other than the specific list of decls that are homed, is the compilation model: MSVC PCH modules are built into the object file for some other source file (when compiling that source file /Yc is specified to say "this compilation is where the PCH is homed"), whereas modular code generation invokes a separate compilation for the PCH alone. So the current modular code generation test of to decide if a decl should be emitted "is the module where this decl is serialized the current main file" has to be extended (as Lubos did in D69778) to also test the command line flag -building-pch-with-obj.

Otherwise the whole thing is basically streamlined down to the modular code generation path.

This even offers one extra material improvement compared to the existing divergent implementation: Homed functions are not emitted into object files that use the pch. Instead at -O0 they are not emitted into the IR at all, and at -O1 they are emitted using available_externally (existing functionality implemented for modular code generation). The pch-codegen test has been updated to reflect this new behavior.

[If possible: I'd love it if we could not have the extra MSVC-style way of accessing dllexport-pch-homing, and just do it the modular codegen way, but I understand that it might be a limitation of existing build systems. @hans / @thakis: Do either of you know if it'd be practical to move to something more similar to .pcm handling, where the pch itself is passed to the compilation, rather than homed as a side effect of compiling some other source file?]

Reviewers: llunak, hans

Differential Revision: https://reviews.llvm.org/D83652

show more ...

recommit 1b978ddba05c [CUDA][HIP][OpenMP] Emit deferred diagnostics by a post-parsing AST travese

Differential Revision: https://reviews.llvm.org/D70172

[clang] Improve LLVM-style RTTI support in ExternalASTSource/ExternalSemaSource

Summary:
We currently have some very basic LLVM-style RTTI support in the ExternalASTSource class hierarchy
based on t

[clang] Improve LLVM-style RTTI support in ExternalASTSource/ExternalSemaSource

Summary:
We currently have some very basic LLVM-style RTTI support in the ExternalASTSource class hierarchy
based on the `SemaSource` bool( to discriminate it form the ExternalSemaSource). As ExternalASTSource
is supposed to be subclassed we should have extendable LLVM-style RTTI in this class hierarchy to make life easier
for projects building on top of Clang.

Most notably the current RTTI implementation forces LLDB to implement RTTI for its
own ExternalASTSource class (ClangExternalASTSourceCommon) by keeping a global set of
ExternalASTSources that are known to be ClangExternalASTSourceCommon. Projects
using Clang currently have to dosimilar workarounds to get RTTI support for their subclasses.

This patch turns this into full-fledged LLVM-style RTTI based on a static `ID` variable similar to
other LLVM class hierarchies. Also removes the friend declaration from ExternalASTSource to
its child class that was only used to grant access to the `SemaSource` member.

Reviewers: aprantl, dblaikie, rjmccall

Reviewed By: aprantl

Subscribers: riccibruno, labath, lhames, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71397

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Remove trailing space

sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h}

llvm-svn: 338291

[clang-cl] Don't emit dllexport inline functions etc. from pch files (PR37801)

With MSVC, PCH files are created along with an object file that needs to
be linked into the final library or executable

[clang-cl] Don't emit dllexport inline functions etc. from pch files (PR37801)

With MSVC, PCH files are created along with an object file that needs to
be linked into the final library or executable. That object file
contains the code generated when building the headers. In particular, it
will include definitions of inline dllexport functions, and because they
are emitted in this object file, other files using the PCH do not need
to emit them. See the bug for an example.

This patch makes clang-cl match MSVC's behaviour in this regard, causing
significant compile-time savings when building dlls using precompiled
headers.

For example, in a 64-bit optimized shared library build of Chromium with
PCH, it reduces the binary size and compile time of
stroke_opacity_custom.obj from 9315564 bytes to 3659629 bytes and 14.6
to 6.63 s. The wall-clock time of building blink_core.dll goes from
38m41s to 22m33s. ("user" time goes from 1979m to 1142m).

Differential Revision: https://reviews.llvm.org/D48426

llvm-svn: 335466

show more ...