[clang][serialization] Blobify IMPORTS strings and signatures (#116095)

This PR changes a part of the PCM format to store string-like things in
the blob attached to a record instead of VBR6-encodin

[clang][serialization] Blobify IMPORTS strings and signatures (#116095)

This PR changes a part of the PCM format to store string-like things in
the blob attached to a record instead of VBR6-encoding them into the
record itself. Applied to the `IMPORTS` section (which is very hot),
this speeds up dependency scanning by 2.8%.

show more ...

[Serialization] Avoid repeated hash lookups (NFC) (#110950)

[Serialization] No transitive identifier change (#92085)

Following of https://github.com/llvm/llvm-project/pull/92083

The motivation is still cutting of the unnecessary change in the
dependency

[Serialization] No transitive identifier change (#92085)

Following of https://github.com/llvm/llvm-project/pull/92083

The motivation is still cutting of the unnecessary change in the
dependency chain. See the above link (recursively) for details.

After this patch, (and the above patch), we can already do something
pretty interesting. For example,

#### Motivation example

```

//--- m-partA.cppm
export module m:partA;

export inline int getA() {
return 43;
}

export class A {
public:
int getMem();
};

export template <typename T>
class ATempl {
public:
T getT();
};

//--- m-partA.v1.cppm
export module m:partA;

export inline int getA() {
return 43;
}

// Now we add a new declaration without introducing a new type.
// The consuming module which didn't use m:partA completely is expected to be
// not changed.
export inline int getA2() {
return 88;
}

export class A {
public:
int getMem();
// Now we add a new declaration without introducing a new type.
// The consuming module which didn't use m:partA completely is expected to be
// not changed.
int getMem2();
};

export template <typename T>
class ATempl {
public:
T getT();
// Add a new declaration without introducing a new type.
T getT2();
};

//--- m-partB.cppm
export module m:partB;

export inline int getB() {
return 430;
}

//--- m.cppm
export module m;
export import :partA;
export import :partB;

//--- useBOnly.cppm
export module useBOnly;
import m;

export inline int get() {
return getB();
}
```

In this example, module `m` exports two partitions `:partA` and
`:partB`. And a consumer `useBOnly` only consumes the entities from
`:partB`. So we don't hope the BMI of `useBOnly` changes if only
`:partA` changes. After this patch, we can make it if the change of
`:partA` doesn't introduce new types. (And we can get rid of this if we
make no-transitive-type-change).

As the example shows, when we change the implementation of `:partA` from
`m-partA.cppm` to `m-partA.v1.cppm`, we add new function declaration
`getA2()` at the global namespace, add a new member function `getMem2()`
to class `A` and add a new member function to `getT2()` to class
template `ATempl`. And since `:partA` is not used by `useBOnly`
completely, the BMI of `useBOnly` won't change after we made above
changes.

#### Design details

Method used in this patch is similar with
https://github.com/llvm/llvm-project/pull/92083 and
https://github.com/llvm/llvm-project/pull/86912. It extends the 32 bit
IdentifierID to 64 bits and use the higher 32 bits to store the module
file index. So that the encoding of the identifier won't get affected by
other modules.

#### Overhead

Similar with https://github.com/llvm/llvm-project/pull/92083 and
https://github.com/llvm/llvm-project/pull/86912. The change is only
expected to increase the size of the on-disk .pcm files and not affect
the compile-time performances. And from my experiment, the size of the
on-disk change only increase 1%+ and observe no compile-time impacts.

#### Future Plans

I'll try to do the same thing for type ids. IIRC, it won't change the
dependency graph if we add a new type in an unused units. I do think
this is a significant win. And this will be a pretty good answer to "why
modules are better than headers."

show more ...

[unused-includes][Serialization] Remove unused includes. NFC. (#88790)

[clang] Drop unaligned from calls to readNext (NFC) (#88842)

Now readNext defaults to unaligned accesses. This patch drops
unaligned to improve readability.

[clang] NFCI: Make `ModuleFile::File` non-optional (#74892)

AFAICT, `ModuleFile::File` can be `std::nullopt` only for PCM files
loaded from the standard input. This patch starts setting that variab

[clang] NFCI: Make `ModuleFile::File` non-optional (#74892)

AFAICT, `ModuleFile::File` can be `std::nullopt` only for PCM files
loaded from the standard input. This patch starts setting that variable
to `FileManager::getSTDIN()` in that case, which makes it possible to
remove the optionality, and also simplifies code that actually reads the
file.

This is part of an effort to get rid of
`Optional{File,Directory}EntryRefDegradesTo{File,Directory}EntryPtr`.

show more ...

Use llvm::endianness::{big,little,native} (NFC)

Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class. This patch replaces
{big,little,native} with ll

Use llvm::endianness::{big,little,native} (NFC)

Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class. This patch replaces
{big,little,native} with llvm::endianness::{big,little,native}.

This patch completes the migration to llvm::endianness and
llvm::endianness::{big,little,native}. I'll post a separate patch to
remove the migration helpers in llvm/Support/Endian.h:

using endianness = llvm::endianness;
constexpr llvm::endianness big = llvm::endianness::big;
constexpr llvm::endianness little = llvm::endianness::little;
constexpr llvm::endianness native = llvm::endianness::native;

show more ...

[clang] NFCI: Use `FileEntryRef` in `GlobalModuleIndex`

Reland "[clang][modules] Move `UNHASHED_CONTROL_BLOCK` up in the AST file"

This reverts commit b6ba804f7775f89f230ee1e62526a2f8225c7966, effectively relanding commit 7d1565727dad3acb54fe76a908630843

Reland "[clang][modules] Move `UNHASHED_CONTROL_BLOCK` up in the AST file"

This reverts commit b6ba804f7775f89f230ee1e62526a2f8225c7966, effectively relanding commit 7d1565727dad3acb54fe76a908630843835d7bc8.

The original commit incorrectly called `ASTWriter::writeUnhashedControlBlock()` before `ASTWriter::collectNonAffectingInputFiles()`, causing SourceLocations/FileIDs in the pragma diagnostic mappings block to be invalid. This is now tested by `clang/test/Modules/diag-mappings-affecting.c`.

show more ...

Revert "[clang][modules] Move `UNHASHED_CONTROL_BLOCK` up in the AST file"

This reverts commit 7d1565727dad3acb54fe76a908630843835d7bc8.

[clang][modules] Move `UNHASHED_CONTROL_BLOCK` up in the AST file

When loading (transitively) imported AST file, `ModuleManager::addModule()` first checks it has the expected signature via `readASTF

[clang][modules] Move `UNHASHED_CONTROL_BLOCK` up in the AST file

When loading (transitively) imported AST file, `ModuleManager::addModule()` first checks it has the expected signature via `readASTFileSignature()`. The signature is part of `UNHASHED_CONTROL_BLOCK`, which is placed at the end of the AST file. This means that just to verify signature of an AST file, we need to skip over all top-level blocks, paging in the whole AST file from disk. This is pretty slow.

This patch moves `UNHASHED_CONTROL_BLOCK` to the start of the AST file, so that it can be read more efficiently. To achieve this, we use dummy signature when first emitting the unhashed control block, and then backpatch the real signature at the end of the serialization process.

This speeds up dependency scanning by over 9% and significantly reduces run-to-run variability of my benchmarks.

Depends on D158572.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D158573

show more ...

[clang] NFC, replace llvm::writeFileAtomically with llvm::writeToOutput API inGlobalModuleIndex.cpp

We're in favor of writeToOutput.

[Serialization] Remove unused function getKnownModules

The last use was removed by:

commit 603cd869f7cdb0da7a545e86a1786f3175f72475
Author: Douglas Gregor <dgregor@apple.com>
Date: Fri Mar

[Serialization] Remove unused function getKnownModules

The last use was removed by:

commit 603cd869f7cdb0da7a545e86a1786f3175f72475
Author: Douglas Gregor <dgregor@apple.com>
Date: Fri Mar 22 18:50:14 2013 +0000

show more ...

[C++20] [Modules] Emit a warning if the we load the modules by implicit generated path

A step to address https://github.com/llvm/llvm-project/issues/62707.

It is not user friendly enough to drop th

[C++20] [Modules] Emit a warning if the we load the modules by implicit generated path

A step to address https://github.com/llvm/llvm-project/issues/62707.

It is not user friendly enough to drop the implicitly generated path
directly. Let's emit the warning first and drop it in the next version.

show more ...

Undo a bit of fcf4e360ba6b that confuses MSVC

clang\lib\Serialization\GlobalModuleIndex.cpp(818): error C2440: 'initializing': cannot convert from 'const ValueTy' to '_Ty2 &&'
with
[

Undo a bit of fcf4e360ba6b that confuses MSVC

clang\lib\Serialization\GlobalModuleIndex.cpp(818): error C2440: 'initializing': cannot convert from 'const ValueTy' to '_Ty2 &&'
with
[
ValueTy=llvm::SmallVector<unsigned int,2>
]
and
[
_Ty2=llvm::SmallVector<unsigned int,2>
]

show more ...

Iterate over StringMaps using structured bindings. NFCI.

[Clang] Use std::move in GlobalModuleIndex::readIndex. NFC

BitstreamCursors are heavy-weight objects that should not be passed by value.

Differential Revision: https://reviews.llvm.org/D123436

[clang][NFC] Use SmallString instead of SmallVector<char

Simplifies code in some places and is more explicit about what is being used.
No additional includes were added here so no impact on compile

[clang][NFC] Use SmallString instead of SmallVector<char

Simplifies code in some places and is more explicit about what is being used.
No additional includes were added here so no impact on compile time.

show more ...

Return false if the identifier is not in the global module index.

This allows clients to use the idiom:

if (GlobalIndex->lookupIdentifier(Name, FoundModules)) {
// work on the FoundModules
}

Thi

Return false if the identifier is not in the global module index.

This allows clients to use the idiom:

if (GlobalIndex->lookupIdentifier(Name, FoundModules)) {
// work on the FoundModules
}

This is also a minor performance improvent for clang.

Differential Revision: https://reviews.llvm.org/D81077

show more ...

Make ASTFileSignature an array of 20 uint8_t instead of 5 uint32_t

Reviewers: aprantl, dexonsmith, Bigcheese

Subscribers: arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews

Make ASTFileSignature an array of 20 uint8_t instead of 5 uint32_t

Reviewers: aprantl, dexonsmith, Bigcheese

Subscribers: arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81347

show more ...

GlobalModuleIndex - Fix use-after-move clang static analyzer warning.

Shadow variable names meant we were referencing the Buffer input argument, not the GlobalModuleIndex member that its std::move()

GlobalModuleIndex - Fix use-after-move clang static analyzer warning.

Shadow variable names meant we were referencing the Buffer input argument, not the GlobalModuleIndex member that its std::move()'d it.

show more ...

[Support] Add TimeTraceScope constructor without detail arg

This simplifies code where no extra details are required
Also don't write out detail when it is empty.

Differential Revision: https://rev

[Support] Add TimeTraceScope constructor without detail arg

This simplifies code where no extra details are required
Also don't write out detail when it is empty.

Differential Revision: https://reviews.llvm.org/D71347

show more ...

clang/Modules: Move Serialization/Module.{h,cpp} to ModuleFile, NFC

Remove some cognitive load by renaming clang/Serialization/Module.h to
clang/Serialization/ModuleFile.h, since it declares the Mod

clang/Modules: Move Serialization/Module.{h,cpp} to ModuleFile, NFC

Remove some cognitive load by renaming clang/Serialization/Module.h to
clang/Serialization/ModuleFile.h, since it declares the ModuleFile
class. This also makes editing a bit easier, since the basename of the
file no long conflicts with clang/Basic/Module.h, which declares the
Module class. Also move lib/Serialization/Module.cpp to
lib/Serialization/ModuleFile.cpp.

show more ...

[Support] Add overload writeFileAtomically(std::function Writer)

Differential Revision: https://reviews.llvm.org/D67424

llvm-svn: 371890

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan Haskins <harlan@apple.com>
llvm-svn: 367616

show more ...