[VPlan] Add helper to run VPlan passes, verify after run (NFC). (#123640)

Add new runPass helpers to run a VPlan transformation. This makes it
easier to add additional checks/functionality for each

[VPlan] Add helper to run VPlan passes, verify after run (NFC). (#123640)

Add new runPass helpers to run a VPlan transformation. This makes it
easier to add additional checks/functionality for each transform run. In
this patch, an option is added to run the verifier after each VPlan
transform.

Follow-ups will use the same helper to also support printing VPlans
after each transform.

Note that the verifier at the moment requires there to be a canonical IV
and vector loop region, so the final lowering transforms aren't run via
runPass yet.

PR: https://github.com/llvm/llvm-project/pull/123640

show more ...

[NFC][DebugInfo] Switch more call-sites to using iterator-insertion (#124283)

To finalise the "RemoveDIs" work removing debug intrinsics, we're
updating call sites that insert instructions to use i

[NFC][DebugInfo] Switch more call-sites to using iterator-insertion (#124283)

To finalise the "RemoveDIs" work removing debug intrinsics, we're
updating call sites that insert instructions to use iterators instead.
This set of changes are those where it's not immediately obvious that
just calling getIterator to fetch an iterator is correct, and one or two
places where more than one line needs to change.

Overall the same rule holds though: iterators generated for the start of
a block such as getFirstNonPHIIt need to be passed into insert/move
methods without being unwrapped/rewrapped, everything else can use
getIterator.

show more ...

[VPlan] Refactor HCFG builder to preserve original vector latch (NFC).

Update HCFG builder to preserve the original latch block of the initial
VPlan, ensuring there is always a latch.

It also skips

[VPlan] Refactor HCFG builder to preserve original vector latch (NFC).

Update HCFG builder to preserve the original latch block of the initial
VPlan, ensuring there is always a latch.

It also skips creating the BranchOnCond for the latch of the top-level
loop, instead of removing it later. Exiting via the latch is controlled
by later recipes.

This further unifies HCFG construction and prepares for use to also
build an initial VPlan (VPlan0) for inner loops.

show more ...

[NFC][DebugInfo] Use iterator-flavour getFirstNonPHI at many call-sites (#123737)

As part of the "RemoveDIs" project, BasicBlock::iterator now carries a
debug-info bit that's needed when getFirstNo

[NFC][DebugInfo] Use iterator-flavour getFirstNonPHI at many call-sites (#123737)

As part of the "RemoveDIs" project, BasicBlock::iterator now carries a
debug-info bit that's needed when getFirstNonPHI and similar feed into
instruction insertion positions. Call-sites where that's necessary were
updated a year ago; but to ensure some type safety however, we'd like to
have all calls to getFirstNonPHI use the iterator-returning version.

This patch changes a bunch of call-sites calling getFirstNonPHI to use
getFirstNonPHIIt, which returns an iterator. All these call sites are
where it's obviously safe to fetch the iterator then dereference it. A
follow-up patch will contain less-obviously-safe changes.

We'll eventually deprecate and remove the instruction-pointer
getFirstNonPHI, but not before adding concise documentation of what
considerations are needed (very few).

---------

Co-authored-by: Stephen Tozer <Melamoto@gmail.com>

show more ...

[NFC][DebugInfo] Use iterator moveBefore at many call-sites (#123583)

As part of the "RemoveDIs" project, BasicBlock::iterator now carries a
debug-info bit that's needed when getFirstNonPHI and sim

[NFC][DebugInfo] Use iterator moveBefore at many call-sites (#123583)

As part of the "RemoveDIs" project, BasicBlock::iterator now carries a
debug-info bit that's needed when getFirstNonPHI and similar feed into
instruction insertion positions. Call-sites where that's necessary were
updated a year ago; but to ensure some type safety however, we'd like to
have all calls to moveBefore use iterators.

This patch adds a (guaranteed dereferenceable) iterator-taking
moveBefore, and changes a bunch of call-sites where it's obviously safe
to change to use it by just calling getIterator() on an instruction
pointer. A follow-up patch will contain less-obviously-safe changes.

We'll eventually deprecate and remove the instruction-pointer
insertBefore, but not before adding concise documentation of what
considerations are needed (very few).

show more ...

[LoopVectorize][NFC] Centralize the setting of CostKind (#121937)

In each class which calculates instruction costs (VPCostContext,
LoopVectorizationCostModel, GeneratedRTChecks) set the CostKind on

[LoopVectorize][NFC] Centralize the setting of CostKind (#121937)

In each class which calculates instruction costs (VPCostContext,
LoopVectorizationCostModel, GeneratedRTChecks) set the CostKind once in
the constructor instead of in each function that calculates a cost. This
is in preparation for potentially changing the CostKind when compiling
for optsize.

show more ...

[VPlan][Coverity] Fix coverity CID1579964. (#121805)

Fix for the Coverity hit with CID1579964 in VPlan.cpp.

Coverity message with some context follows.

[Cov] var_compare_op: Comparing TermBr t

[VPlan][Coverity] Fix coverity CID1579964. (#121805)

Fix for the Coverity hit with CID1579964 in VPlan.cpp.

Coverity message with some context follows.

[Cov] var_compare_op: Comparing TermBr to null implies that TermBr might
be null.
434 } else if (TermBr && !TermBr->isConditional()) {
435 TermBr->setSuccessor(0, NewBB);
436 } else {
437 // Set each forward successor here when it is created, excluding
438 // backedges. A backward successor is set when the branch is
created.
439 unsigned idx = PredVPSuccessors.front() == this ? 0 : 1;

[Cov] CID 1579964: (#1 of 1): Dereference after null check
(FORWARD_NULL)
[Cov] var_deref_model: Passing null pointer TermBr to getSuccessor,
which dereferences it.

show more ...

[VPlan] Remove loop region in optimizeForVFAndUF. (#108378)

Update optimizeForVFAndUF to completely remove the vector loop region
when possible. At the moment, we cannot remove the region if it con

[VPlan] Remove loop region in optimizeForVFAndUF. (#108378)

Update optimizeForVFAndUF to completely remove the vector loop region
when possible. At the moment, we cannot remove the region if it contains

* widened IVs: the recipe is needed to generate the step vector
* reductions: ComputeReductionResults requires the reduction phi recipe
for codegen.

Both cases can be addressed by more explicit modeling.

The patch also includes a number of updates to allow executing VPlans
without a vector loop region.

Depends on https://github.com/llvm/llvm-project/pull/110004

show more ...

[VPlan] Remove re-using vector PH in VPBasicBlock::execute (NFC).

Remove logic to re-use the previous basic block for the vector pre
header from VPBasicBlock::execute. The preheader is now modeled a

[VPlan] Remove re-using vector PH in VPBasicBlock::execute (NFC).

Remove logic to re-use the previous basic block for the vector pre
header from VPBasicBlock::execute. The preheader is now modeled as
VPIRBasicBlock, so the code is no longer needed.

Split off from https://github.com/llvm/llvm-project/pull/108378.

show more ...

[VPlan] Replace VPBBs with VPIRBBs during skeleton creation (NFC).

Move replacement of VPBBs for vector preheader, middle block and scalar
preheader from VPlan::execute to skeleton creation, which a

[VPlan] Replace VPBBs with VPIRBBs during skeleton creation (NFC).

Move replacement of VPBBs for vector preheader, middle block and scalar
preheader from VPlan::execute to skeleton creation, which actually
creates the IR basic blocks.

For now, the vector preheader can only be replaced after
prepareToExecute as it may create new instructions in the vector
preheader.

show more ...

[VPlan] Add all blocks to outer loop if present during ::execute (NFCI).

This ensures that all blocks created during VPlan execution are properly
added to an enclosing loop, if present.

Split off f

[VPlan] Add all blocks to outer loop if present during ::execute (NFCI).

This ensures that all blocks created during VPlan execution are properly
added to an enclosing loop, if present.

Split off from https://github.com/llvm/llvm-project/pull/108378 and also
needed once more of the skeleton blocks are created directly via VPlan.

This also allows removing the custom logic for early-exit loop
vectorization added as part of
https://github.com/llvm/llvm-project/pull/117008.

show more ...

[VPlan] Manage created blocks directly in VPlan. (NFC) (#120918)

This patch changes the way blocks are managed by VPlan. Previously all
blocks reachable from entry would be cleaned up when a VPlan

[VPlan] Manage created blocks directly in VPlan. (NFC) (#120918)

This patch changes the way blocks are managed by VPlan. Previously all
blocks reachable from entry would be cleaned up when a VPlan is
destroyed. With this patch, each VPlan keeps track of blocks created for
it in a list and this list is then used to delete all blocks in the list
when the VPlan is destroyed. To do so, block creation is funneled
through helpers in directly in VPlan.

The main advantage of doing so is it simplifies CFG transformations, as
those do not have to take care of deleting any blocks, just adjusting
the CFG. This helps to simplify
https://github.com/llvm/llvm-project/pull/108378 and
https://github.com/llvm/llvm-project/pull/106748.

This also simplifies handling of 'immutable' blocks a VPlan holds
references to, which at the moment only include the scalar header block.

PR: https://github.com/llvm/llvm-project/pull/120918

show more ...

[VPlan] Remove redundant printing final in VPlan::execute (#121048)

Multiple prints will cause problems when testing ir-bb

[VPlan] Move initial VPlan block creation to constructor. (NFC)

This sets up the initial blocks needed to initialize a VPlan directly
in the constructor. This will allow tracking of all created bloc

[VPlan] Move initial VPlan block creation to constructor. (NFC)

This sets up the initial blocks needed to initialize a VPlan directly
in the constructor. This will allow tracking of all created blocks
directly in VPlan, simplifying block deletion.

show more ...

[VPlan] Remove legacy VPlan() constructors (NFC).

The constructors were retained to reduce the diff during transition.

Remove them now.

[VPlan] Add VPWidenInduction recipe as common base class (NFC). (#120008)

This helps to simplify some existing code and new code
(https://github.com/llvm/llvm-project/pull/112145)

PR: https://gi

[VPlan] Add VPWidenInduction recipe as common base class (NFC). (#120008)

This helps to simplify some existing code and new code
(https://github.com/llvm/llvm-project/pull/112145)

PR: https://github.com/llvm/llvm-project/pull/120008

show more ...

[VPlan] Move ::getVectorLoopRegion out of ifdef (NFC).

Fixes a build failure with assertions disabled after
6c8f41d336747.

[VPlan] Hook IR blocks into VPlan during skeleton creation (NFC) (#114292)

As a first step to move towards modeling the full skeleton in VPlan,
start by wrapping IR blocks created during legacy ske

[VPlan] Hook IR blocks into VPlan during skeleton creation (NFC) (#114292)

As a first step to move towards modeling the full skeleton in VPlan,
start by wrapping IR blocks created during legacy skeleton creation in
VPIRBasicBlocks and hook them into the VPlan. This means the skeleton
CFG is represented in VPlan, just before execute. This allows moving
parts of skeleton creation into recipes in the VPBBs gradually.

Note that this allows retiring some manual DT updates, as this will be
handled automatically during VPlan execution.

PR: https://github.com/llvm/llvm-project/pull/114292

show more ...

[VPlan] Use variadic isa<> in a few more places. NFC (#119538)

[VPlan] Dispatch to multiple exit blocks via middle blocks. (#112138)

A more lightweight variant of
https://github.com/llvm/llvm-project/pull/109193,
which dispatches to multiple exit blocks via t

[VPlan] Dispatch to multiple exit blocks via middle blocks. (#112138)

A more lightweight variant of
https://github.com/llvm/llvm-project/pull/109193,
which dispatches to multiple exit blocks via the middle blocks.

The patch also introduces a bit of required scaffolding to enable
early-exit vectorization, including an option. At the moment, early-exit
vectorization doesn't come with legality checks, and is only used if the
option is provided and the loop has metadata forcing vectorization. This
is only intended to be used for testing during bring-up, with @david-arm
enabling auto early-exit vectorization plugging in the changes from
https://github.com/llvm/llvm-project/pull/88385.

PR: https://github.com/llvm/llvm-project/pull/112138

show more ...

[VPlan] Move convertToConreteRecipes to end of VPlan-opt phase (NFCI).

Adjust placement as suggested in
https://github.com/llvm/llvm-project/pull/114305, after some refactoring
to prepare for the mo

[VPlan] Move convertToConreteRecipes to end of VPlan-opt phase (NFCI).

Adjust placement as suggested in
https://github.com/llvm/llvm-project/pull/114305, after some refactoring
to prepare for the move.

show more ...

[VPlan] Create canonical IV resume value for epilogue in VPlan. (NFCI)

Update the code to create induction resume PHIs to also create a resume
phi for the canonical induction during epilogue vectori

[VPlan] Create canonical IV resume value for epilogue in VPlan. (NFCI)

Update the code to create induction resume PHIs to also create a resume
phi for the canonical induction during epilogue vectorization. This
unifies the code for handling induction resume values and removes the
need to explicitly create manually resume PHI and return it during
epilogue creation.

Overall it helps to move the code for updating the canonical induction
resume value to the place where all other header phi resume values are
updated.

This is NFC, modulo order of the created phis.

show more ...

[VPlan] Iterate over blocks in VPlan::execute in RPOT (NFC).

This prepares for more complex CFGs in VPlan, as in
https://github.com/llvm/llvm-project/pull/114292
https://github.com/llvm/llvm

[VPlan] Iterate over blocks in VPlan::execute in RPOT (NFC).

This prepares for more complex CFGs in VPlan, as in
https://github.com/llvm/llvm-project/pull/114292
https://github.com/llvm/llvm-project/pull/112138

show more ...

[VPlan] Move printing final VPlan to ::execute (NFC).

This moves printing of the final VPlan to ::execute. This ensures the
final VPlan is printed, including recipes that get introduced by late,
low

[VPlan] Move printing final VPlan to ::execute (NFC).

This moves printing of the final VPlan to ::execute. This ensures the
final VPlan is printed, including recipes that get introduced by late,
lowering transforms and skeleton construction.

Split off from https://github.com/llvm/llvm-project/pull/114292, to
simplify the diff.

show more ...

[VPlan] Use RPOT for VPlan codegen and printing.

This split off changes for more complex CFGs in VPlan from both
https://github.com/llvm/llvm-project/pull/114292
https://github.com/llvm/llvm

[VPlan] Use RPOT for VPlan codegen and printing.

This split off changes for more complex CFGs in VPlan from both
https://github.com/llvm/llvm-project/pull/114292
https://github.com/llvm/llvm-project/pull/112138

This simplifies their respective diffs.

show more ...