Reland "[llvm-cov] Support multi-source object files for convert-for-testing"

`llvm-cov convert-for-testing` only functions properly when the input binary contains a single source file. When the bin

Reland "[llvm-cov] Support multi-source object files for convert-for-testing"

`llvm-cov convert-for-testing` only functions properly when the input binary contains a single source file. When the binary has multiple source files, a `Malformed coverage data` error will occur when the generated .covmapping is read back. This is because the testing format lacks support for indicating the size of its file records, and current implementation just assumes there's only one record in it. This patch fixes this problem by introducing a new testing format version.

Changes to the code:

- Add a new format version. The version number is stored in the the last 8 bytes of the orignial magic number field to be backward-compatible.
- Output a LEB128 number before the file records section to indicate its size in the new version.
- Change the format parsing code correspondingly.
- Update the document to formalize the testing format.
- Additionally, fix the bug when converting COFF binaries.

Reviewed By: phosek, gulfem

Differential Revision: https://reviews.llvm.org/D156611

show more ...

Revert "[llvm-cov] Fix a bug about using `convert-for-testing` on multi-source object files"

This reverts commit f8ad86c23405168a8cd189590184fdcc296627e0 because
it broke some downsteam tests report

Revert "[llvm-cov] Fix a bug about using `convert-for-testing` on multi-source object files"

This reverts commit f8ad86c23405168a8cd189590184fdcc296627e0 because
it broke some downsteam tests reported in https://reviews.llvm.org/D156611.

show more ...

[llvm-cov] Fix a bug about using `convert-for-testing` on multi-source object files

`llvm-cov convert-for-testing` is used to build the .covmapping files used in its regression tests. However the cu

[llvm-cov] Fix a bug about using `convert-for-testing` on multi-source object files

`llvm-cov convert-for-testing` is used to build the .covmapping files used in its regression tests. However the current implementation only works when there's only one source file in the mapping information data. If there are more than 1 source files, `llvm-cov convert-for-testing` can still produce a .covmapping file, but when read it back, `llvm-cov` will report:

```
error: Failed to load coverage: 'main.covmapping': Malformed coverage data
```

This is because the output .covmapping file doesn't have any mark to indicate the boundary between file records and function records, and current implementation jsut assume there's only one file record in the .covmapping file.

Changes to the code:

- Make `llvm-cov convert-for-testing` output a LEB128 number before file records to indicate its size.
- Change the testing format parsing code correspondingly.
- Update existing .covmapping files.

Reviewed By: gulfem

Differential Revision: https://reviews.llvm.org/D156611

show more ...

[Object] Refactor build ID parsing into Object lib.

This makes parsing for build IDs in the markup filter slightly more
permissive, in line with fromHex.

It also removes the distinction between mis

[Object] Refactor build ID parsing into Object lib.

This makes parsing for build IDs in the markup filter slightly more
permissive, in line with fromHex.

It also removes the distinction between missing build ID and empty build
ID; empty build IDs aren't a useful concept, since their purpose is to
uniquely identify a binary. This removes a layer of indirection wherever
build IDs are obtained.

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D147485

show more ...

[NFC][TargetParser] Remove llvm/ADT/Triple.h

I also ran `git clang-format` to get the headers in the right order for
the new location, which has changed the order of other headers in two
files.

Reland: [llvm-cov] Look up object files using debuginfod

Reviewed By: gulfem

Differential Revision: https://reviews.llvm.org/D136702

Revert "[llvm-cov] Look up object files using debuginfod"

This reverts commit efbc8bb18eda63007216ad0cb5a8de04963eddd5.

This change is causing failures when detecting curl on several build bots:
-

Revert "[llvm-cov] Look up object files using debuginfod"

This reverts commit efbc8bb18eda63007216ad0cb5a8de04963eddd5.

This change is causing failures when detecting curl on several build bots:
- https://lab.llvm.org/buildbot/#/builders/247/builds/884
- https://lab.llvm.org/buildbot/#/builders/231/builds/7688
- https://lab.llvm.org/buildbot/#/builders/121/builds/27389
- https://lab.llvm.org/buildbot/#/builders/230/builds/8464
- https://lab.llvm.org/buildbot/#/builders/57/builds/24209
- https://lab.llvm.org/buildbot/#/builders/127/builds/42722

show more ...

[llvm-cov] Look up object files using debuginfod

Reviewed By: gulfem

Differential Revision: https://reviews.llvm.org/D136702

Move from llvm::makeArrayRef to ArrayRef deduction guides - llvm/ part

Use deduction guides instead of helper functions.

The only non-automatic changes have been:

1. ArrayRef(some_uint8_pointer, 0

Move from llvm::makeArrayRef to ArrayRef deduction guides - llvm/ part

Use deduction guides instead of helper functions.

The only non-automatic changes have been:

1. ArrayRef(some_uint8_pointer, 0) needs to be changed into ArrayRef(some_uint8_pointer, (size_t)0) to avoid an ambiguous call with ArrayRef((uint8_t*), (uint8_t*))
2. CVSymbol sym(makeArrayRef(symStorage)); needed to be rewritten as CVSymbol sym{ArrayRef(symStorage)}; otherwise the compiler is confused and thinks we have a (bad) function prototype. There was a few similar situation across the codebase.
3. ADL doesn't seem to work the same for deduction-guides and functions, so at some point the llvm namespace must be explicitly stated.
4. The "reference mode" of makeArrayRef(ArrayRef<T> &) that acts as no-op is not supported (a constructor cannot achieve that).

Per reviewers' comment, some useless makeArrayRef have been removed in the process.

This is a follow-up to https://reviews.llvm.org/D140896 that introduced
the deduction guides.

Differential Revision: https://reviews.llvm.org/D140955

show more ...

[ProfileData] llvm::Optional => std::optional

[llvm] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of ma

[llvm] Use std::nullopt instead of None (NFC)

This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[Support] Rename llvm::compression::{zlib,zstd}::uncompress to more appropriate decompress

This improves consistency with other places (e.g. llvm::compression::decompress,
llvm::object::Decompressor

[Support] Rename llvm::compression::{zlib,zstd}::uncompress to more appropriate decompress

This improves consistency with other places (e.g. llvm::compression::decompress,
llvm::object::Decompressor::decompress, llvm-objcopy).
Note: when zstd::uncompress was added, we noticed that the API `ZSTD_decompress`
is fine while the zlib API `uncompress` is a misnomer.

show more ...

[Support] Change compression::zlib::{compress,uncompress} to use uint8_t *

It's more natural to use uint8_t * (std::byte needs C++17 and llvm has
too much uint8_t *) and most callers use uint8_t * i

[Support] Change compression::zlib::{compress,uncompress} to use uint8_t *

It's more natural to use uint8_t * (std::byte needs C++17 and llvm has
too much uint8_t *) and most callers use uint8_t * instead of char *.
The functions are recently moved into `llvm::compression::zlib::`, so
downstream projects need to make adaption anyway.

show more ...

[NFC] Refactor llvm::zlib namespace

* Refactor compression namespaces across the project, making way for a possible
introduction of alternatives to zlib compression.
Changes are as follows:
*

[NFC] Refactor llvm::zlib namespace

* Refactor compression namespaces across the project, making way for a possible
introduction of alternatives to zlib compression.
Changes are as follows:
* Relocate the `llvm::zlib` namespace to `llvm::compression::zlib`.

Reviewed By: MaskRay, leonardchan, phosek

Differential Revision: https://reviews.llvm.org/D128953

show more ...

[CoverageMapping] Remove dots from paths inside the profile

We already remove dots from collected paths and path mappings. This
makes it difficult to match paths inside the profile which contain
dot

[CoverageMapping] Remove dots from paths inside the profile

We already remove dots from collected paths and path mappings. This
makes it difficult to match paths inside the profile which contain
dots. For example, we would never match /path/to/../file.c because
the collected path is always be normalized to /path/file.c. This
change enables dot removal for paths inside the profile to address
the issue.

Differential Revision: https://reviews.llvm.org/D123164

show more ...

Revert "[CoverageMapping] Remove dots from paths inside the profile"

This reverts commit d1b098fc825176242afee12b8f9dc14adf5eec51 since
it is failing on Windows builders.

[CoverageMapping] Remove dots from paths inside the profile

We already remove dots from collected paths and path mappings. This
makes it difficult to match paths inside the profile which contain
dot

[CoverageMapping] Remove dots from paths inside the profile

We already remove dots from collected paths and path mappings. This
makes it difficult to match paths inside the profile which contain
dots. For example, we would never match /path/to/../file.c because
the collected path is always be normalized to /path/file.c. This
change enables dot removal for paths inside the profile to address
the issue.

Differential Revision: https://reviews.llvm.org/D122750

show more ...

Cleanup LLVMObject headers

Most notably,

llvm/Object/Binary.h no longer includes llvm/Support/MemoryBuffer.h
llvm/Object/MachOUniversal*.h no longer include llvm/Object/Archive.h
llvm/Object/TapiUn

Cleanup LLVMObject headers

Most notably,

llvm/Object/Binary.h no longer includes llvm/Support/MemoryBuffer.h
llvm/Object/MachOUniversal*.h no longer include llvm/Object/Archive.h
llvm/Object/TapiUniversal.h no longer includes llvm/Object/TapiFile.h

llvm-project preprocessed size:
before: 1068185081
after: 1068324320

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D119457

show more ...

[LLVM][NFC]Inclusive language: remove occurances of sanity check/test from llvm

Part of work to use more inclusive language in clang/llvm. Rewording
some comments and change function and variable na

[LLVM][NFC]Inclusive language: remove occurances of sanity check/test from llvm

Part of work to use more inclusive language in clang/llvm. Rewording
some comments and change function and variable names.

show more ...

[compiler-rt] Add more diagnostic to InstrProfError

If profile data is malformed for any kind of reason, we generate
an error that only reports "malformed instrumentation profile data"
without any f

[compiler-rt] Add more diagnostic to InstrProfError

If profile data is malformed for any kind of reason, we generate
an error that only reports "malformed instrumentation profile data"
without any further information. This patch extends InstrProfError
class to receive an optional error message argument, so that we can
do better error reporting.

Differential Revision: https://reviews.llvm.org/D108942

show more ...

[llvm-cov] Enforce alignment of function records

Function Records are required to be aligned on 8 bytes. This is enforced for each
records except the first, when one relies on the default alignment

[llvm-cov] Enforce alignment of function records

Function Records are required to be aligned on 8 bytes. This is enforced for each
records except the first, when one relies on the default alignment within an
std::string. There's no such guarantee, and indeed on 32 bits for some
implementation of std::string this is not enforced.

Provide a portable implementation based on llvm's MemoryBuffer.

Differential Revision: https://reviews.llvm.org/D104745

show more ...

[Coverage] Support overriding compilation directory

When making compilation relocatable, for example in distributed
compilation scenarios, we want to set compilation dir to a relative
value like `.`

[Coverage] Support overriding compilation directory

When making compilation relocatable, for example in distributed
compilation scenarios, we want to set compilation dir to a relative
value like `.` but this presents a problem when generating reports
because if the file path is relative as well, for example `..`, you
may end up writing files outside of the output directory.

This change introduces a flag that allows overriding the compilation
directory that's stored inside the profile with a different value that
is absolute.

Differential Revision: https://reviews.llvm.org/D100232

show more ...

[llvm-cov] Support for v4 format in convert-for-testing

v4 moves function records to a dedicated section so we need to write
and read it separately.

https://reviews.llvm.org/D100535

[Support] Don't include VirtualFileSystem.h in CommandLine.h

CommandLine.h is indirectly included in ~50% of TUs when building
clang, and VirtualFileSystem.h is large.

(Already remarked by jhenders

[Support] Don't include VirtualFileSystem.h in CommandLine.h

CommandLine.h is indirectly included in ~50% of TUs when building
clang, and VirtualFileSystem.h is large.

(Already remarked by jhenderson on D70769.)

No behavior change.

Differential Revision: https://reviews.llvm.org/D100957

show more ...

[Coverage] Store compilation dir separately in coverage mapping

We currently always store absolute filenames in coverage mapping. This
is problematic for several reasons. It poses a problem for dis

[Coverage] Store compilation dir separately in coverage mapping

We currently always store absolute filenames in coverage mapping. This
is problematic for several reasons. It poses a problem for distributed
compilation as source location might vary across machines. We are also
duplicating the path prefix potentially wasting space.

This change modifies how we store filenames in coverage mapping. Rather
than absolute paths, it stores the compilation directory and file paths
as given to the compiler, either relative or absolute. Later when
reading the coverage mapping information, we recombine relative paths
with the working directory. This approach is similar to handling
ofDW_AT_comp_dir in DWARF.

Finally, we also provide a new option, -fprofile-compilation-dir akin
to -fdebug-compilation-dir which can be used to manually override the
compilation directory which is useful in distributed compilation cases.

Differential Revision: https://reviews.llvm.org/D95753

show more ...