[SampleFDO] New hierarchical discriminator for Flow Sensitive SampleFDO

This patch implements first part of Flow Sensitive SampleFDO (FSAFDO).
It has the following changes:
(1) disable current discr

[SampleFDO] New hierarchical discriminator for Flow Sensitive SampleFDO

This patch implements first part of Flow Sensitive SampleFDO (FSAFDO).
It has the following changes:
(1) disable current discriminator encoding scheme,
(2) new hierarchical discriminator for FSAFDO.

For this patch, option "-enable-fs-discriminator=true" turns on the new
functionality. Option "-enable-fs-discriminator=false" (the default)
keeps the current SampleFDO behavior. When the fs-discriminator is
enabled, we insert a flag variable, namely, llvm_fs_discriminator, to
the object. This symbol will checked by create_llvm_prof tool, and used
to generate a profile with FS-AFDO discriminators enabled. If this
happens, for an extbinary format profile, create_llvm_prof tool
will add a flag to profile summary section.

Differential Revision: https://reviews.llvm.org/D102246

show more ...

[X86] Support AMX fast register allocation

Differential Revision: https://reviews.llvm.org/D100026

Revert "[X86] Support AMX fast register allocation"

This reverts commit 77e2e5e07d01fe0b83c39d0c527c0d3d2e659146.

[X86] Support AMX fast register allocation

Recommit "[VP,Integer,#2] ExpandVectorPredication pass"

This reverts the revert 02c5ba8679873e878ae7a76fb26808a47940275b

Fix:

Pass was registered as DUMMY_FUNCTION_PASS causing the newpm-pass
func

Recommit "[VP,Integer,#2] ExpandVectorPredication pass"

This reverts the revert 02c5ba8679873e878ae7a76fb26808a47940275b

Fix:

Pass was registered as DUMMY_FUNCTION_PASS causing the newpm-pass
functions to be doubly defined. Triggered in -DLLVM_ENABLE_MODULE=1
builds.

Original commit:

This patch implements expansion of llvm.vp.* intrinsics
(https://llvm.org/docs/LangRef.html#vector-predication-intrinsics).

VP expansion is required for targets that do not implement VP code
generation. Since expansion is controllable with TTI, targets can switch
on the VP intrinsics they do support in their backend offering a smooth
transition strategy for VP code generation (VE, RISC-V V, ARM SVE,
AVX512, ..).

Reviewed By: rogfer01

Differential Revision: https://reviews.llvm.org/D78203

show more ...

Revert "[VP,Integer,#2] ExpandVectorPredication pass"

This reverts commit 43bc584dc05e24c6d44ece8e07d4bff585adaf6d.

The commit broke the -DLLVM_ENABLE_MODULES=1 builds.

http://green.lab.llvm.org/g

Revert "[VP,Integer,#2] ExpandVectorPredication pass"

This reverts commit 43bc584dc05e24c6d44ece8e07d4bff585adaf6d.

The commit broke the -DLLVM_ENABLE_MODULES=1 builds.

http://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/31603/consoleFull#2136199809a1ca8a51-895e-46c6-af87-ce24fa4cd561

show more ...

[VP,Integer,#2] ExpandVectorPredication pass

This patch implements expansion of llvm.vp.* intrinsics
(https://llvm.org/docs/LangRef.html#vector-predication-intrinsics).

VP expansion is required for

[VP,Integer,#2] ExpandVectorPredication pass

This patch implements expansion of llvm.vp.* intrinsics
(https://llvm.org/docs/LangRef.html#vector-predication-intrinsics).

VP expansion is required for targets that do not implement VP code
generation. Since expansion is controllable with TTI, targets can switch
on the VP intrinsics they do support in their backend offering a smooth
transition strategy for VP code generation (VE, RISC-V V, ARM SVE,
AVX512, ..).

Reviewed By: rogfer01

Differential Revision: https://reviews.llvm.org/D78203

show more ...

Revert "[X86] Support AMX fast register allocation"

This reverts commit 3b8ec86fd576b9808dc63da620d9a4f7bbe04372.

Revert "[X86] Refine AMX fast register allocation"

This reverts commit c3f95e91976

Revert "[X86] Support AMX fast register allocation"

This reverts commit 3b8ec86fd576b9808dc63da620d9a4f7bbe04372.

Revert "[X86] Refine AMX fast register allocation"

This reverts commit c3f95e9197643b699b891ca416ce7d72cf89f5fc.

This pass breaks using LLVM in a multi-threaded environment by
introducing global state.

show more ...

[X86] Support AMX fast register allocation

Differential Revision: https://reviews.llvm.org/D100026

Revert "Remove "Rewrite Symbols" from codegen pipeline"

This reverts commit 6210261ecb21c84c9a440a76c0ccbc8ad211bed3.

addr-label.ll crashes on armv7.

Remove "Rewrite Symbols" from codegen pipeline

It breaks up the function pass manager in the codegen pipeline.

With empty parameters, it looks at the -mllvm flag -rewrite-map-file.
This is likely n

Remove "Rewrite Symbols" from codegen pipeline

It breaks up the function pass manager in the codegen pipeline.

With empty parameters, it looks at the -mllvm flag -rewrite-map-file.
This is likely not in use.

Add a check that we only have one function pass manager in the codegen
pipeline.

This required reverting commit 9583a3f2625818b78c0cf6d473cdedb9f23ad82c:
"[AsmPrinter] Delete dead takeDeletedSymbsForFunction()".
This was not NFC as initially thought. By coalescing two function
psas managers, this exposed the reverted code as necessary.
addr-label.ll was crashing due to an emitted blockaddress's block being
removed but the label not emitted.

Some tests relied on the fact that we had a module pass somewhere in the
codegen pipeline.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D99707

show more ...

Move EntryExitInstrumentation pass location

This seems to be more of a Clang thing rather than a generic LLVM thing,
so this moves it out of LLVM pipelines and as Clang extension hooks into
LLVM pip

Move EntryExitInstrumentation pass location

This seems to be more of a Clang thing rather than a generic LLVM thing,
so this moves it out of LLVM pipelines and as Clang extension hooks into
LLVM pipelines.

Move the post-inline EEInstrumentation out of the backend pipeline and
into a late pass, similar to other sanitizer passes. It doesn't fit
into the codegen pipeline.

Also fix up EntryExitInstrumentation not running at -O0 under the new
PM. PR49143

Reviewed By: hans

Differential Revision: https://reviews.llvm.org/D97608

show more ...

[CodeGen] New pass: Replace vector intrinsics with call to vector library

This patch adds a pass to replace calls to vector intrinsics (i.e., LLVM
intrinsics operating on vector operands) with calls

[CodeGen] New pass: Replace vector intrinsics with call to vector library

This patch adds a pass to replace calls to vector intrinsics (i.e., LLVM
intrinsics operating on vector operands) with calls to a vector library.

Currently, calls to LLVM intrinsics are only replaced with calls to vector
libraries when scalar calls to intrinsics are vectorized by the Loop- or
SLP-Vectorizer.

With this pass, it is now possible to replace calls to LLVM intrinsics
already operating on vector operands, e.g., if such code was generated
by MLIR. For the replacement, information from the TargetLibraryInfo,
e.g., as specified via -vector-library is used.

This is a re-try of the original commit 2303e93e66 that was reverted
due to pass manager problems. Other minor changes have also been made.

Differential Revision: https://reviews.llvm.org/D95373

show more ...

[CodeGen] Basic block sections should take precendence over splitting.

The use of basic block sections should take precedence over the machine
function splitting pass. Since they use the same underl

[CodeGen] Basic block sections should take precendence over splitting.

The use of basic block sections should take precedence over the machine
function splitting pass. Since they use the same underlying mechanism
they are kept exclusive. Updated the tests to check that split machine
functions is overridden by all flavours of basic block sections.

Differential Revision: https://reviews.llvm.org/D96392

show more ...

Revert "[Codegen][ReplaceWithVecLib] add pass to replace vector intrinsics with calls to vector library"

This reverts commit 2303e93e666e13ebf6d24323729c28f520ecca37.
Investigating bot failures.

[Codegen][ReplaceWithVecLib] add pass to replace vector intrinsics with calls to vector library

This patch adds a pass to replace calls to vector intrinsics
(i.e., LLVM intrinsics operating on vecto

[Codegen][ReplaceWithVecLib] add pass to replace vector intrinsics with calls to vector library

This patch adds a pass to replace calls to vector intrinsics
(i.e., LLVM intrinsics operating on vector operands) with
calls to a vector library.

Currently, calls to LLVM intrinsics are only replaced with
calls to vector libraries when scalar calls to intrinsics are
vectorized by the Loop- or SLP-Vectorizer.

With this pass, it is now possible to replace calls to LLVM
intrinsics already operating on vector operands, e.g., if
such code was generated by MLIR. For the replacement,
information from the TargetLibraryInfo, e.g., as specified
via -vector-library is used.

Differential Revision: https://reviews.llvm.org/D95373

show more ...

CodeGen: Refactor regallocator command line and target selection

Make the sequence of passes to select and rewrite instructions to
physical registers be a target callback. This is to prepare to allo

CodeGen: Refactor regallocator command line and target selection

Make the sequence of passes to select and rewrite instructions to
physical registers be a target callback. This is to prepare to allow
targets to split register allocation into multiple phases.

show more ...

Reland "[NewPM][CodeGen] Introduce CodeGenPassBuilder to help build codegen pipeline" (again)

This reverts commit 16c8f6e91344ec9840d6aa9ec6b8d0c87a104ca3 with fix.

-Wswitch catched an unhandled en

Reland "[NewPM][CodeGen] Introduce CodeGenPassBuilder to help build codegen pipeline" (again)

This reverts commit 16c8f6e91344ec9840d6aa9ec6b8d0c87a104ca3 with fix.

-Wswitch catched an unhandled enum value due to recent commits in
TargetPassConfig.cpp.

show more ...

Revert "Reland "[NewPM][CodeGen] Introduce CodeGenPassBuilder to help build codegen pipeline""

This reverts commit 21314940c4856e0cb81b664fd2d2117d1b7dc3e3.

Build failure in some bots.

Reland "[NewPM][CodeGen] Introduce CodeGenPassBuilder to help build codegen pipeline"

This reverts commit 94427af60c66ffea655a3084825c6c3a9deec1ad (relands
4646de5d75cfce3da4ddeffb6eb8e66e38238800 w

Reland "[NewPM][CodeGen] Introduce CodeGenPassBuilder to help build codegen pipeline"

This reverts commit 94427af60c66ffea655a3084825c6c3a9deec1ad (relands
4646de5d75cfce3da4ddeffb6eb8e66e38238800 with fix).

Use "return std::move(AsmStreamer);" instead of "return AsmStreamer;" in
LVMTargetMachine::createMCStreamer. Unlike Clang, GCC seems having trouble
inserting a implicit lvalue->rvalue conversion.

show more ...

Revert "[NewPM][CodeGen] Introduce CodeGenPassBuilder to help build codegen pipeline"

This reverts commit 4646de5d75cfce3da4ddeffb6eb8e66e38238800.

Some bots have build failure.

[NewPM][CodeGen] Introduce CodeGenPassBuilder to help build codegen pipeline

Following up on D67687.
Please refer to the RFC here http://lists.llvm.org/pipermail/llvm-dev/2020-July/143309.html

`Cod

[NewPM][CodeGen] Introduce CodeGenPassBuilder to help build codegen pipeline

Following up on D67687.
Please refer to the RFC here http://lists.llvm.org/pipermail/llvm-dev/2020-July/143309.html

`CodeGenPassBuilder` is the NPM counterpart of `TargetPassConfig` with below differences.
- Debugging features (MIR print/verify, disable pass, start/stop-before/after, etc.) living in `TargetPassConfig` are moved to use PassInstrument as much as possible. (Implementation also lives in `TargetPassConfig.cpp`)
- `TargetPassConfig` is a polymorphic base (virtual inheritance) to build the target-dependent pipeline whereas `CodeGenPassBuilder` is the CRTP base/helper to implement the target-dependent pipeline. The motivation is flexibility for targets to customize the pipeline, inlining opportunity, and fits the overall NPM value semantics design.
- `TargetPassConfig` is a legacy immutable pass to declare hooks for targets to customize some target-independent codegen layer behavior. This is partially ported to TargetMachine::options. The rest, such as `createMachineScheduler/createPostMachineScheduler`, are left out for now. They should be implemented in LLVMTargetMachine in the future.

Reviewed By: arsenm, aeubanks

Differential Revision: https://reviews.llvm.org/D83608

show more ...

[Debugify] Support checking Machine IR debug info
Add mir-check-debug pass to check MIR-level debug info.

For IR-level, currently, LLVM have debugify + check-debugify to generate
and check debug IR.

[Debugify] Support checking Machine IR debug info
Add mir-check-debug pass to check MIR-level debug info.

For IR-level, currently, LLVM have debugify + check-debugify to generate
and check debug IR. Much like the IR-level pass debugify, mir-debugify
inserts sequentially increasing line locations to each MachineInstr in a
Module, But there is no equivalent MIR-level check-debugify pass, So now
we support it at "mir-check-debug".

Reviewed By: djtodoro

Differential Revision: https://reviews.llvm.org/D91595

show more ...

Revert "[Debugify] Support checking Machine IR debug info"

This reverts commit 50aaa8c274910d78d7bf6c929a34fe58b1f45579.

[Debugify] Support checking Machine IR debug info
Add mir-check-debug pass to check MIR-level debug info.

For IR-level, currently, LLVM have debugify + check-debugify to generate
and check debug IR.

[Debugify] Support checking Machine IR debug info
Add mir-check-debug pass to check MIR-level debug info.

For IR-level, currently, LLVM have debugify + check-debugify to generate
and check debug IR. Much like the IR-level pass debugify, mir-debugify
inserts sequentially increasing line locations to each MachineInstr in a
Module, But there is no equivalent MIR-level check-debugify pass, So now
we support it at "mir-check-debug".

Reviewed By: djtodoro

Differential Revision: https://reviews.llvm.org/D91595

show more ...