Fix to D139603(reverted) - moved size check to unit test so that it is cross-platform

D139603 (add option to llvm-profdata to reduce output profile size) contains test cases that are not cross-platf

Fix to D139603(reverted) - moved size check to unit test so that it is cross-platform

D139603 (add option to llvm-profdata to reduce output profile size) contains test cases that are not cross-platform. Moving those tests to unit test and making sure the feature is callable from llvm library

Reviewed By: snehasish

Differential Revision: https://reviews.llvm.org/D141446

show more ...

[llvm-profdata] Remove an unused include after D115915

Revert "[llvm-profdata] Add option to cap profile output size"

This reverts commit 5b72d0e4f5eeb8f90c744cac8e0728cffeca61a9.

The test added is failing on Mac/Windows. See review for buildbot failur

Revert "[llvm-profdata] Add option to cap profile output size"

This reverts commit 5b72d0e4f5eeb8f90c744cac8e0728cffeca61a9.

The test added is failing on Mac/Windows. See review for buildbot failure links.

show more ...

[llvm-profdata] Add option to cap profile output size

Allow user to specify `--output-size-limit=n` to cap the size of generated profile to be strictly under n. Functions with the lowest total sampl

[llvm-profdata] Add option to cap profile output size

Allow user to specify `--output-size-limit=n` to cap the size of generated profile to be strictly under n. Functions with the lowest total sample count are dropped first if necessary. Due to using a heuristic, excessive functions may be dropped to satisfy the size requirement

Reviewed By: snehasish

Differential Revision: https://reviews.llvm.org/D139603

show more ...

[profile] Add binary ids into indexed profiles

This patch adds support for including binary ids in an indexed profile.
It adds a new field into the header that points to the offset of the
binary id

[profile] Add binary ids into indexed profiles

This patch adds support for including binary ids in an indexed profile.
It adds a new field into the header that points to the offset of the
binary id section. The binary id section consists of a size of the
section, and a list of binary ids (if they are present) that consist
of two parts: length and data.

This patch guarantees that indexed profile is backwards compatible
after adding binary ids.

Differential Revision: https://reviews.llvm.org/D135929

show more ...

Revert "[profile] Add binary ids into indexed profiles"

This reverts commit 7734053fd98e7d5ddc749808ce38134686425fb7
because it broke powerpc64 bot:
https://lab.llvm.org/buildbot#builders/231/builds

Revert "[profile] Add binary ids into indexed profiles"

This reverts commit 7734053fd98e7d5ddc749808ce38134686425fb7
because it broke powerpc64 bot:
https://lab.llvm.org/buildbot#builders/231/builds/6229

show more ...

[profile] Add binary ids into indexed profiles

This patch adds support for including binary ids in an indexed profile.
It adds a new field into the header that points to the offset of the
binary id

[profile] Add binary ids into indexed profiles

This patch adds support for including binary ids in an indexed profile.
It adds a new field into the header that points to the offset of the
binary id section. The binary id section consists of a size of the
section, and a list of binary ids (if they are present) that consist
of two parts: length and data.

This patch guarantees that indexed profile is backwards compatible
after adding binary ids.

Differential Revision: https://reviews.llvm.org/D135929

show more ...

[InstrProf] Fix bug when merging empty profile with multiple threads

When merging profiles with multiple threads, the `mergeWriterContexts()` function is used to merge profile data between writers.

[InstrProf] Fix bug when merging empty profile with multiple threads

When merging profiles with multiple threads, the `mergeWriterContexts()` function is used to merge profile data between writers. This must be in sync with `loadInput()` which merges profiles to a single writer. This diff merges the profile kind correctly in `mergeWriterContexts()` to fix a subtle bug.

Reviewed By: phosek

Differential Revision: https://reviews.llvm.org/D139755

show more ...

[llvm-profdata] Drop profile symbol list during merging AutoFDO profiles.

Adding a switch to drop profile symbol list during merging AutoFDO profiles. This is needed to minimize the impact on defaul

[llvm-profdata] Drop profile symbol list during merging AutoFDO profiles.

Adding a switch to drop profile symbol list during merging AutoFDO profiles. This is needed to minimize the impact on default profiles when the profile symbol list is enabled for the source input profiles. The symbol list is quite large and could potentially slow down the compiler.

Reviewed By: davidxl, wenlei

Differential Revision: https://reviews.llvm.org/D139486

show more ...

[llvm-profdata] Use flattening sample profile in profile supplementation

We need to flatten the SampleFDO profile in profile supplementation
because the InstrFDO profile does not have inlined callsi

[llvm-profdata] Use flattening sample profile in profile supplementation

We need to flatten the SampleFDO profile in profile supplementation
because the InstrFDO profile does not have inlined callsite counters.
Without flattening profile, FDO optimizations are not stable:
we will not supplement the second generation profile when the modified
functions are all inlined.

This patch fixes this issue: we will flatten the profile for functions
that appears in FDO profile.

Note that we only need to find the hot/warm functions in SampleFDO
profile, so we will not perform a full flatten. We will use
a DFS traversal to compute the accumulated entry count and max bodycount.
This is much cheaper than full flattening.

Differential Revision: https://reviews.llvm.org/D138893

show more ...

[llvm-profdata] Use std::optional in llvm-profdata.cpp (NFC)

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasv

[llvm-profdata] Use std::optional in llvm-profdata.cpp (NFC)

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716

show more ...

[llvm-profdata] Rename show flag to --show-format

In https://reviews.llvm.org/D135127 we created the show flag
`--output-format` which was confusing because it behaved differently
than the same flag

[llvm-profdata] Rename show flag to --show-format

In https://reviews.llvm.org/D135127 we created the show flag
`--output-format` which was confusing because it behaved differently
than the same flag in the merge command. So, rename the flag to
`--show-format`. This also allows us to add the `text` option to mean
"normal text output" rather than "text-encoded profiles" like it does
for the merge command.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D135467

show more ...

[InstrProf][llvm-profdata] Dump profile correlation data as YAML

Change the behavior of the `llvm-profdata show --debug-info=` command to dump a YAML file when using debug info correlation since it

[InstrProf][llvm-profdata] Dump profile correlation data as YAML

Change the behavior of the `llvm-profdata show --debug-info=` command to dump a YAML file when using debug info correlation since it provides more information in a parseable format.

Reviewed By: yozhu, phosek

Differential Revision: https://reviews.llvm.org/D134770

show more ...

[llvm-profdata] Add --output-format option

Add `--output-format` option for the `llvm-profdata show` command to select the type of output. The existing `--text` flag is used to emit text encoded pro

[llvm-profdata] Add --output-format option

Add `--output-format` option for the `llvm-profdata show` command to select the type of output. The existing `--text` flag is used to emit text encoded profiles. To avoid confusion, `--output-format=text-encoding` indicates that the output will be profiles encoded in the text format, and `--output-format=text` indicates the default text output that doesn't necessarily represent a profile.
`--output-format=json` is an alias for `--json` and `--output-format=yaml` will be used in D134770.

Reviewed By: phosek

Differential Revision: https://reviews.llvm.org/D135127

show more ...

[InstrProf] Add version into llvm-profdata

This patch adds support of printing profile version
into llvm-profdata which was proposed in:
https://discourse.llvm.org/t/llvm-profdata-failure-guarantees

[InstrProf] Add version into llvm-profdata

This patch adds support of printing profile version
into llvm-profdata which was proposed in:
https://discourse.llvm.org/t/llvm-profdata-failure-guarantees-for-code-coverage/64924

Differential Revision: https://reviews.llvm.org/D135317

show more ...

[llvm-driver] Add various tools to the llvm-driver

The llvm-driver, enabled with LLVM_TOOL_LLVM_DRIVER_BUILD combines many llvm executables
into one to save overall toolchain size. This patch adds a

[llvm-driver] Add various tools to the llvm-driver

The llvm-driver, enabled with LLVM_TOOL_LLVM_DRIVER_BUILD combines many llvm executables
into one to save overall toolchain size. This patch adds a few more llvm tools to the
llvm-driver.

Differential Revision: https://reviews.llvm.org/D135281

show more ...

[iwyu] Move <cmath> out of llvm/Support/MathExtras.h

Interestingly, MathExtras.h doesn't use <cmath> declaration, so move it out of
that header and include it when needed.

No functional change inte

[iwyu] Move <cmath> out of llvm/Support/MathExtras.h

Interestingly, MathExtras.h doesn't use <cmath> declaration, so move it out of
that header and include it when needed.

No functional change intended, but there's no longer a transitive include
fromMathExtras.h to cmath.

show more ...

[llvm-profdata] Improve profile supplementation

Current implementation promotes a non-cold function in the SampleFDO profile
into a hot function in the FDO profile. This is too aggressive. This patc

[llvm-profdata] Improve profile supplementation

Current implementation promotes a non-cold function in the SampleFDO profile
into a hot function in the FDO profile. This is too aggressive. This patch
promotes a hot functions in the SampleFDO profile into a hot function, and a
warm function in SampleFDO into a warm function in FDO.

Differential Revision: https://reviews.llvm.org/D132601

show more ...

[llvm-profdata] Handle internal linkage functions in profile supplementation

This patch has the following changes:
(1) Handling of internal linkage functions (static functions)
Static functions in F

[llvm-profdata] Handle internal linkage functions in profile supplementation

This patch has the following changes:
(1) Handling of internal linkage functions (static functions)
Static functions in FDO have a prefix of source file name, while they do not
have one in SampleFDO. Current implementation does not handle this and we are
not updating the profile for static functions. This patch fixes this.

(2) Handling of -funique-internal-linakge-symbols
Again this is for the internal linkage functions. Option
-funique-internal-linakge-symbols can now be applied to both FDO and SampleFDO
compilation. When it is used, it demangles internal linkage function names and
adds a hash value as the postfix.

When both SampleFDO and FDO profiles use this option, or both
not use this option, changes in (1) should handle this.

Here we also handle when the SampleFDO profile using this option while FDO
profile not using this option, or vice versa.

There is one case where this patch won't work: If one of the profiles used
mangled name and the other does not. For example, if the SampleFDO profile
uses clang c-compiler and without -funique-internal-linakge-symbols, while
the FDO profile uses -funique-internal-linakge-symbols. The SampleFDO profile
contains unmangled names while the FDO profile contains mangled names. If
both profiles use c++ compiler, this won't happen. We think this use case
is rare and does not justify the effort to fix.

Differential Revision: https://reviews.llvm.org/D132600

show more ...

[llvm-profdata] Adjust profile supplementation heuristics

1) We now use the count size in FDO as the main factor to deal with pre-inliner.
Currently we use the number of sample records in the Sample

[llvm-profdata] Adjust profile supplementation heuristics

1) We now use the count size in FDO as the main factor to deal with pre-inliner.
Currently we use the number of sample records in the SampleFDO profile. But
that only counts the top-level body sample records (not including the nested
call-sites). We are seeing some big functions not being updated because of
this. I think using the count size in FDO profile is more reasonable to judge if
the function is likely to be inlined to the callers in pre-inliner.

(2) We use getMaxCount in SampleFDO rather the HeadSample to determine if
if the function is hot in SampleFDO. This is in-sync with the logic
in the compiler (also HeadSample can be 0).

Differential Revision: https://reviews.llvm.org/D132602

show more ...

[llvm-profdata] Support JSON as as an output-only format

This patch teaches llvm-profdata to output the sample profile in the
JSON format. The new option is intended to be used for research and
dev

[llvm-profdata] Support JSON as as an output-only format

This patch teaches llvm-profdata to output the sample profile in the
JSON format. The new option is intended to be used for research and
development purposes. For example, one can write a Python script to
take a JSON file and analyze how similar different inline instances of
a given function are to each other.

I've chosen JSON because Python can parse it reasonably fast, and it
just takes a couple of lines to read the whole data:

import json
with open ('profile.json') as f:
profile = json.load(f)

Differential Revision: https://reviews.llvm.org/D130944

show more ...

Use any_of (NFC)

[NFC] FunctionSamples::getEntrySamples -> getHeadSamplesEstimate

The name `getEntrySamples` was misleading for 2 reasons. One, it's
close in name to `Function::getEntryCount`, but the equivalent her

[NFC] FunctionSamples::getEntrySamples -> getHeadSamplesEstimate

The name `getEntrySamples` was misleading for 2 reasons. One, it's
close in name to `Function::getEntryCount`, but the equivalent here is
`getHeadSamples`; second, as opposed to the other get* APIs in
`FunctionSamples`, it performs an estimate/heuristic rather than just
retrieving raw data (or a non-heuristic derivate off that data, like
`getMaxCountInside`)

The new name should more clearly communicate its intent; and, being
close (in name) to `getHeadSamples`, it should allow the reader discover
the relation between them.

Also updated the doc comments for both `getHeadSamples[Estimate]` so a
reader may better understand the relation between them.

Differential Revision: https://reviews.llvm.org/D130281

show more ...

[llvm] Don't use Optional::hasValue (NFC)

This patch replaces Optional::hasValue with the implicit cast to bool
in conditionals only.

Revert "Don't use Optional::hasValue (NFC)"

This reverts commit aa8feeefd3ac6c78ee8f67bf033976fc7d68bc6d.