[BOLT] Introduce binary analysis tool based on BOLT (#115330)

This initial commit does not add any specific binary analyses yet, it
merely contains the boilerplate to introduce a new BOLT-based too

[BOLT] Introduce binary analysis tool based on BOLT (#115330)

This initial commit does not add any specific binary analyses yet, it
merely contains the boilerplate to introduce a new BOLT-based tool.

This basically combines the 4 first patches from the prototype pac-ret
and stack-clash binary analyzer discussed in RFC
https://discourse.llvm.org/t/rfc-bolt-based-binary-analysis-tool-to-verify-correctness-of-security-hardening/78148
and published at
https://github.com/llvm/llvm-project/compare/main...kbeyls:llvm-project:bolt-gadget-scanner-prototype

The introduction of such a BOLT-based binary analysis tool was proposed
and discussed in at least the following places:
- The RFC pointed to above
- EuroLLVM 2024 round table
https://discourse.llvm.org/t/summary-of-bolt-as-a-binary-analysis-tool-round-table-at-eurollvm/78441
The round table showed quite a few people interested in being able to
build a custom binary analysis quickly with a tool like this.
- Also at the US LLVM dev meeting a few weeks ago, I heard interest from
a few people, asking when the tool would be available upstream.
- The presentation "Adding Pointer Authentication ABI support for your
ELF platform"
(https://llvm.swoogo.com/2024devmtg/session/2512720/adding-pointer-authentication-abi-support-for-your-elf-platform)
explicitly mentioned interest to extend the prototype tool to verify
correct implementation of pauthabi.

show more ...

[BOLT] Enable standalone build (#97130)

Continue from #87196 as author did not have much time, I have taken over
working on this PR. We would like to have this so it'll be easier to
package for Ni

[BOLT] Enable standalone build (#97130)

Continue from #87196 as author did not have much time, I have taken over
working on this PR. We would like to have this so it'll be easier to
package for Nix.

Can be tested by copying cmake, bolt, third-party, and llvm directories
out into their own directory with this PR applied and then build bolt.

---------

Co-authored-by: pca006132 <john.lck40@gmail.com>

show more ...

[BOLT][Instrumentation] Add test for append-pid option

Reviewed By: rafauler

Differential Revision: https://reviews.llvm.org/D154121

[NFC][Py Reformat] Reformat python files in the rest of the dirs

This is an ongoing series of commits that are reformatting our
Python code. This catches the last of the python files to
reformat. Si

[NFC][Py Reformat] Reformat python files in the rest of the dirs

This is an ongoing series of commits that are reformatting our
Python code. This catches the last of the python files to
reformat. Since they where so few I bunched them together.

Reformatting is done with `black`.

If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.

If you run into any problems, post to discourse about it and
we will try to help.

RFC Thread below:

https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style

Reviewed By: jhenderson, #libc, Mordante, sivachandra

Differential Revision: https://reviews.llvm.org/D150784

show more ...

[BOLT] Use MCInstPrinter in createRetpolineFunctionTag

Make retpoline functions invariant of X86 register numbers.
retpoline-synthetic.test is known to fail NFC testing due to shifting
register numb

[BOLT] Use MCInstPrinter in createRetpolineFunctionTag

Make retpoline functions invariant of X86 register numbers.
retpoline-synthetic.test is known to fail NFC testing due to shifting
register numbers. Use canonical register names instead of tablegen
numbers.

Before:
```
__retpoline_r51_
__retpoline_mem_r58+DATAat0x200fe8
__retpoline_mem_r51+0
__retpoline_mem_r132+0+8*53
```

After:
```
__retpoline_%rax_
__retpoline_mem_%rip+DATAat0x200fe8
__retpoline_mem_%rax+0
__retpoline_mem_%r12+0+8*%rbx
```

Test Plan:
- Revert 67bd3c58c0c7389e39c5a2f4d3b1a30459ccf5b7 that touches X86RegisterInfo.td.
- retpoline-synthetic.test passes in NFC mode with this diff, fails without it.

Reviewed By: #bolt, rafauler

Differential Revision: https://reviews.llvm.org/D150138

show more ...

[BOLT] Fix many tests detected as unsupported

Since D148847, many tests are detected as being unsupported. This is
caused by BOLT_TARGETS_TO_BUILD being ;-separated whereas the previously
used TARGE

[BOLT] Fix many tests detected as unsupported

Since D148847, many tests are detected as being unsupported. This is
caused by BOLT_TARGETS_TO_BUILD being ;-separated whereas the previously
used TARGETS_TO_BUILD is space-separated.

This patch fixes this by creating config.targets lit.cfg.py by splitting
on ';'.

Reviewed By: rafauler

Differential Revision: https://reviews.llvm.org/D149026

show more ...

[BOLT] Process fragment siblings in lite mode, keep lite mode on

In lite mode, include split function fragments to the list of functions to
process even if a fragment has no samples. This is require

[BOLT] Process fragment siblings in lite mode, keep lite mode on

In lite mode, include split function fragments to the list of functions to
process even if a fragment has no samples. This is required to properly
detect and update split jump tables (jump tables that contain pointers to code
in the main and cold fragments).

Reviewed By: #bolt, maksfb

Differential Revision: https://reviews.llvm.org/D140457

show more ...

Add BAT testing framework

This patch refactors BAT to be testable as a library, so we
can have open-source tests on it. This further fixes an issue with
basic blocks that lack a valid input offset,

Add BAT testing framework

This patch refactors BAT to be testable as a library, so we
can have open-source tests on it. This further fixes an issue with
basic blocks that lack a valid input offset, making BAT omit those
when writing translation tables.

Test Plan: new testcases added, new testing tool added (llvm-bat-dump)

Differential Revision: https://reviews.llvm.org/D129382

show more ...

[BOLT][DWARF] Fix TU Index handling for DWARF4/5

When we generate split dwarf with -fdebug-types-section we will have
.debug_types.dwo sections. These go into TU Index when we run llvm-dwp. BOLT was

[BOLT][DWARF] Fix TU Index handling for DWARF4/5

When we generate split dwarf with -fdebug-types-section we will have
.debug_types.dwo sections. These go into TU Index when we run llvm-dwp. BOLT was
not handling DWP input correctly with this section.

Added support for handling DWP with TU Index as an input and output for DWARF4.
Added support for handling DWP with TU Index as an input for DWARF5

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D126087

show more ...

[BOLT] Fix LIT tests on Windows VS2019

Fix newline issue in link_fdata.py, as well as how to call the tool.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D126437

[BOLT][TEST] Remove -gdwarf-4 override from %cflags

As BOLT support for monolithic and split DWARF5 is added, remove DWARF version
override for BOLT tests.

Reviewed By: ayermolo

Differential Revis

[BOLT][TEST] Remove -gdwarf-4 override from %cflags

As BOLT support for monolithic and split DWARF5 is added, remove DWARF version
override for BOLT tests.

Reviewed By: ayermolo

Differential Revision: https://reviews.llvm.org/D125366

show more ...

[BOLT] Add R_AARCH64_PREL16/32/64 relocations support

Reviewed By: yota9, rafauler

Differential Revision: https://reviews.llvm.org/D122294

[BOLT][TEST] Remove -no-pie from cflags/cxxflags

Align with an upstream change D120305 to make PIE the default on linux-gnu.

Add `-no-pie` to tests that require it.

Reviewed By: maksfb, yota9

Dif

[BOLT][TEST] Remove -no-pie from cflags/cxxflags

Align with an upstream change D120305 to make PIE the default on linux-gnu.

Add `-no-pie` to tests that require it.

Reviewed By: maksfb, yota9

Differential Revision: https://reviews.llvm.org/D123329

show more ...

[BOLT] Fix plt relocations symbol match

The bfd linker adds the symbol versioning string to the symbol name in symtab.
Skip the versioning part in order to find the registered PLT function.

Vladisl

[BOLT] Fix plt relocations symbol match

The bfd linker adds the symbol versioning string to the symbol name in symtab.
Skip the versioning part in order to find the registered PLT function.

Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei

Differential Revision: https://reviews.llvm.org/D122039

show more ...

[BOLT][test] Enable cross-target testing

Check for supported target architecture instead of the host arch when
deciding to execute non-runtime tests.

Reviewed By: Amir

Differential Revision: https

[BOLT][test] Enable cross-target testing

Check for supported target architecture instead of the host arch when
deciding to execute non-runtime tests.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D122498

show more ...

[BOLT] Add ld.lld substitution

Register ld.lld substition (tests were failing)

Reviewed By: yota9

Differential Revision: https://reviews.llvm.org/D118738

[BOLT][TEST] Add heatmap.test

Add a basic test for heatmap mode

Reviewed By: maksfb, ayermolo

Differential Revision: https://reviews.llvm.org/D118868

[BOLT] Fix DWARFv5 for aarch64

This patch reverts patch "DWARFv5 default: Switch bolt tests to use
DWARFv4 since Bolt doesn't support v5 yet" and places the -gdwarf-4 flag
to the global cflags confi

[BOLT] Fix DWARFv5 for aarch64

This patch reverts patch "DWARFv5 default: Switch bolt tests to use
DWARFv4 since Bolt doesn't support v5 yet" and places the -gdwarf-4 flag
to the global cflags config file.

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D118283

show more ...

[BOLT][CMAKE] Use BOLT_CLANG_EXE and BOLT_LLD_EXE as is

Add an ability to provide paths that don't match tool name exactly:
e.g. clang-13.
Remove use_lld call that sets up unused extra tools.

Test

[BOLT][CMAKE] Use BOLT_CLANG_EXE and BOLT_LLD_EXE as is

Add an ability to provide paths that don't match tool name exactly:
e.g. clang-13.
Remove use_lld call that sets up unused extra tools.

Test plan:
```
cmake -G Ninja ../llvm-project/llvm -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="bolt" -DBOLT_CLANG_EXE=/usr/bin/clang-13 -DBOLT_LLD_EXE=/usr/bin/lld-13
...
llvm-lit: /data/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using clang: /usr/bin/clang-13
llvm-lit: /data/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using lld: /usr/bin/lld-13

cmake -G Ninja ../llvm-project/llvm -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="bolt;lld" -DBOLT_CLANG_EXE=/usr/bin/clang-13
...
llvm-lit: /data/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using clang: /usr/bin/clang-13
llvm-lit: /data/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using lld: /data/llvm-build2/bin/lld

cmake -G Ninja ../llvm-project/llvm -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="bolt;clang;lld"
...
llvm-lit: /data/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using clang: /data/llvm-build3/bin/clang
llvm-lit: /data/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using lld: /data/llvm-build3/bin/lld
```

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D117446

show more ...

[BOLT][CMAKE] Accept BOLT_CLANG_EXE and BOLT_LLD_EXE

Add CMake options to supply clang and lld binaries for use in check-bolt
instead of requiring the build of clang and lld projects.

Suggested by

[BOLT][CMAKE] Accept BOLT_CLANG_EXE and BOLT_LLD_EXE

Add CMake options to supply clang and lld binaries for use in check-bolt
instead of requiring the build of clang and lld projects.

Suggested by Mehdi Amini in https://lists.llvm.org/pipermail/llvm-dev/2021-December/154426.html

Test Plan:
```
cmake -G Ninja ~/local/llvm-project/llvm \
-DLLVM_TARGETS_TO_BUILD="X86" \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DLLVM_ENABLE_PROJECTS="bolt" \
-DBOLT_CLANG_EXE=~/local/bin/clang \
-DBOLT_LLD_EXE=~/local/bin/lld

ninja check-bolt
...
llvm-lit: /home/aaupov/local/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using clang: /home/aaupov/local/bin/clang
llvm-lit: /home/aaupov/local/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using ld.lld: /home/aaupov/local/bin/ld.lld
llvm-lit: /home/aaupov/local/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using lld-link: /home/aaupov/local/bin/lld-link
llvm-lit: /home/aaupov/local/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using ld64.lld: /home/aaupov/local/bin/ld64.lld
llvm-lit: /home/aaupov/local/llvm-project/llvm/utils/lit/lit/llvm/config.py:436: note: using wasm-ld: /home/aaupov/local/bin/wasm-ld
...
```

Tested all configurations:
- LLVM_ENABLE_PROJECTS="bolt;clang;lld" + no BOLT_*_EXE
- LLVM_ENABLE_PROJECTS="bolt;clang" + BOLT_LLD_EXE
- LLVM_ENABLE_PROJECTS="bolt;lld" + BOLT_CLANG_EXE
- LLVM_ENABLE_PROJECTS="bolt" + BOLT_CLANG_EXE + BOLT_LLD_EXE
- LLVM_ENABLE_PROJECTS="bolt;clang;lld" + BOLT_CLANG_EXE + BOLT_LLD_EXE

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D117061

show more ...

Re-enable Windows build and fix issues

Summary:
Fix missing string header file inclusion and link_fdata find
problem in lit tests. Change root-level tests to require
linux. Re-enable Windows in our

Re-enable Windows build and fix issues

Summary:
Fix missing string header file inclusion and link_fdata find
problem in lit tests. Change root-level tests to require
linux. Re-enable Windows in our root CMakeLists.txt.

(cherry picked from FBD33296290)

show more ...

[BOLT] Don't use ld.lld in tests

Summary: Addressing issue 270.

(cherry picked from FBD33255608)

[BOLT][TEST] Add llvm-boltdiff to build/test requirements

Summary: llvm-boltdiff is required for `runtime/meta-merge-fdata.test`

(cherry picked from FBD32442220)

[BOLT][TEST] Import jump-table-icp.test, update link_fdata script

Summary:
Import the test. The assembly input has three functions with associated fdata.
The old link_fdata.sh script only replaces t

[BOLT][TEST] Import jump-table-icp.test, update link_fdata script

Summary:
Import the test. The assembly input has three functions with associated fdata.
The old link_fdata.sh script only replaces the symbol names with symbol values,
whereas fdata format expects to have symbol offsets against the anchor symbol.
Introduce the link_fdata.py script which is able to parse the input and produce
either an offset or an absolute symbol value.

(cherry picked from FBD32256351)

show more ...

[BOLT][TEST] Add instrumentation test using merge-fdata

Summary:
BOLT meta test using merge-fdata tool.

This tests BOLT instrumentation for a non-trivial binary, running instrumented
binary, and us

[BOLT][TEST] Add instrumentation test using merge-fdata

Summary:
BOLT meta test using merge-fdata tool.

This tests BOLT instrumentation for a non-trivial binary, running instrumented
binary, and using the instrumentation profile for BOLT optimizations.
The results are verified between original, instrumented, and optimized binaries.
Additional tested features: boltdiff mode and merge-fdata for two profiles.

merge-fdata tool is linked with relocs on Linux to support this test.

(cherry picked from FBD32141812)

show more ...