[VPlan] Support multi-exit loops in HCFG builder.

Update HCFG construction to support multi-exit loops. If there is no
unique exit block, map the middle block of the initial plan to the exit
block f

[VPlan] Support multi-exit loops in HCFG builder.

Update HCFG construction to support multi-exit loops. If there is no
unique exit block, map the middle block of the initial plan to the exit
block from the latch.

This further unifies HCFG construction and prepares for use to also
build an initial VPlan (VPlan0) for inner loops.

Effectively NFC as this isn't used on the default code path yet.

show more ...

[VPlan] Refactor HCFG builder to preserve original vector latch (NFC).

Update HCFG builder to preserve the original latch block of the initial
VPlan, ensuring there is always a latch.

It also skips

[VPlan] Refactor HCFG builder to preserve original vector latch (NFC).

Update HCFG builder to preserve the original latch block of the initial
VPlan, ensuring there is always a latch.

It also skips creating the BranchOnCond for the latch of the top-level
loop, instead of removing it later. Exiting via the latch is controlled
by later recipes.

This further unifies HCFG construction and prepares for use to also
build an initial VPlan (VPlan0) for inner loops.

show more ...

[VPlan] Always create initial blocks in constructor (NFC).

Update C++ unit tests to use VPlanTestBase to construct initial VPlan,
using a constructor that creates the VP blocks directly in the
const

[VPlan] Always create initial blocks in constructor (NFC).

Update C++ unit tests to use VPlanTestBase to construct initial VPlan,
using a constructor that creates the VP blocks directly in the
constructor.

Split off from and in preparation for
https://github.com/llvm/llvm-project/pull/120918.

show more ...

[VPlan] Hook IR blocks into VPlan during skeleton creation (NFC) (#114292)

As a first step to move towards modeling the full skeleton in VPlan,
start by wrapping IR blocks created during legacy ske

[VPlan] Hook IR blocks into VPlan during skeleton creation (NFC) (#114292)

As a first step to move towards modeling the full skeleton in VPlan,
start by wrapping IR blocks created during legacy skeleton creation in
VPIRBasicBlocks and hook them into the VPlan. This means the skeleton
CFG is represented in VPlan, just before execute. This allows moving
parts of skeleton creation into recipes in the VPBBs gradually.

Note that this allows retiring some manual DT updates, as this will be
handled automatically during VPlan execution.

PR: https://github.com/llvm/llvm-project/pull/114292

show more ...

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handl

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handled by adding
operands to the wrapped phis in the scalar loop header.

Note that the current version only includes the scalar loop header, no
other loop blocks and also does not wrap it in a region block.

PR: https://github.com/llvm/llvm-project/pull/109975

show more ...

[VPlan] Update unit tests to use getVectorLoopRegion (NFC).

Use getVectorLoopRegion to retrieve the vector loop header instead of
relying on getting the successor of the entry block. This makes sure

[VPlan] Update unit tests to use getVectorLoopRegion (NFC).

Use getVectorLoopRegion to retrieve the vector loop header instead of
relying on getting the successor of the entry block. This makes sure the
code still works correctly as more parts of the skeleton are modeled in
VPlan.

show more ...

[VPlan] Use VPInstruction::Name when assigning names (NFCI).

This slightly improves the printing of VPInstructions. NFC except debug
output.

[VPlan] Add getSCEVExprForVPValue util, use to get trip count SCEV (NFC) (#94464)

Add a new getSCEVExprForVPValue utility which can be used to get a SCEV
expression for a VPValue. The initial imple

[VPlan] Add getSCEVExprForVPValue util, use to get trip count SCEV (NFC) (#94464)

Add a new getSCEVExprForVPValue utility which can be used to get a SCEV
expression for a VPValue. The initial implementation only returns SCEVs
for live-in IR values (by constructing a SCEV based on the live-in IR
value) and VPExpandSCEVRecipe. This is enough to serve its first use,
getting a SCEV for a VPlan's trip count, but will be extended in the
future.

It also removes createTripCountSCEV, as the new helper can be used to
retrieve the SCEV from the VPlan.

PR: https://github.com/llvm/llvm-project/pull/94464

show more ...

[VPlan] Model branch cond to enter scalar epilogue in VPlan. (#92651)

This patch moves branch condition creation to enter the scalar epilogue
loop to VPlan. Modeling the branch in the middle block

[VPlan] Model branch cond to enter scalar epilogue in VPlan. (#92651)

This patch moves branch condition creation to enter the scalar epilogue
loop to VPlan. Modeling the branch in the middle block also requires
modeling the successor blocks. This is done using the recently
introduced VPIRBasicBlock.

Note that the middle.block is still created as part of the skeleton and
then patched in during VPlan execution. Unfortunately the skeleton needs
to create the middle.block early on, as it is also used for induction
resume value creation and is also needed to properly update the
dominator tree during skeleton creation.

After this patch lands, I plan to move induction resume value and phi
node creation in the scalar preheader to VPlan. Once that is done, we
should be able to create the middle.block in VPlan directly.

This is a re-worked version based on the earlier
https://reviews.llvm.org/D150398 and the main change is the use of
VPIRBasicBlock.

Depends on https://github.com/llvm/llvm-project/pull/92525

PR: https://github.com/llvm/llvm-project/pull/92651

show more ...

[VPlan] Use ir-bb prefix for VPIRBasicBlock.

Follow-up to adjust the names and tests after
https://github.com/llvm/llvm-project/pull/93398.

[VPlan] Split VPWidenMemoryInstructionRecipe (NFCI). (#87411)

This patch introduces a new VPWidenMemoryRecipe base class and distinct
sub-classes to model loads and stores.

This is a first step

[VPlan] Split VPWidenMemoryInstructionRecipe (NFCI). (#87411)

This patch introduces a new VPWidenMemoryRecipe base class and distinct
sub-classes to model loads and stores.

This is a first step in an effort to simplify and modularize code
generation for widened loads and stores and enable adding further more
specialized memory recipes.

PR: https://github.com/llvm/llvm-project/pull/87411

show more ...

[VPlan] Rename getVPValueOrAddLiveIn -> getOrAddLiveIn (NFCI).

The helper now only deals with live-ins, clarify the name.

[VPlan] Use opaque pointers in VPlan unit test IR (#69947)

Updates the unit tests for VPlan to use opaque pointers in strings
containing LLVM IR. This is to match the similar adjustments being made

[VPlan] Use opaque pointers in VPlan unit test IR (#69947)

Updates the unit tests for VPlan to use opaque pointers in strings
containing LLVM IR. This is to match the similar adjustments being made
for lit tests to use opaque pointers.

show more ...

[VPlan] Add missing output of live-ins to VPlan dot printing.

Split off live-in printing to VPlan::printLiveIns and use it to print
Live-ins when printing in the DOT format.

[VPlan] Move initial skeleton construction to createInitialVPlan. (NFC)

This patch moves creating the middle VPBBs and an initial empty
vector loop region for the top-level loop to createInitialVPl

[VPlan] Move initial skeleton construction to createInitialVPlan. (NFC)

This patch moves creating the middle VPBBs and an initial empty
vector loop region for the top-level loop to createInitialVPlan.

This consolidates code to create the initial VPlan skeleton and enables
adding other bits outside the main region during initial VPlan
construction. In particular, D150398 will add the exit check & branch to
the middle block.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D158333

show more ...

[VPlan] Use printOperands for VPInstruction.

Use the printOperands for printing VPInstruction's operands to be more
in line with other recipes and ensure consistent printing after D15719.

Also remo

[VPlan] Use printOperands for VPInstruction.

Use the printOperands for printing VPInstruction's operands to be more
in line with other recipes and ensure consistent printing after D15719.

Also removes some stray spaces in print output.

show more ...

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is disconnected
at the moment, as the bypass blocks of the skeleton are not yet modeled
in VPlan.

The preheader block is executed before skeleton creation, so the SCEV
expansion results can be used during skeleton creation. At the moment,
the trip count expression and induction steps are expanded in the new
preheader. The remainder of SCEV expansions will be moved gradually in
the future.

D147965 will update skeleton creation to use the steps expanded in the
pre-header to fix #58811.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147964

show more ...

[VPlan] Remove DeadInsts arg from VPInstructionsToVPRecipes (NFC)

The argument isn't used. VPlan-based dead recipe removal can be used
instead.

Recommit "[VPlan] Unify Value2VPValue and VPExternalDefs maps (NFCI)."

This reverts the revert commit 8c2276f89887d0a27298a1bbbd2181fa54bbb509.

The updated patch re-orders the getDefiningRecipe che

Recommit "[VPlan] Unify Value2VPValue and VPExternalDefs maps (NFCI)."

This reverts the revert commit 8c2276f89887d0a27298a1bbbd2181fa54bbb509.

The updated patch re-orders the getDefiningRecipe check in getVPalue to avoid
a use-after-free.

Original commit message:

Before this patch, a VPlan contained 2 mappings for Values -> VPValue:
1) Value2VPValue and 2) VPExternalDefs.

This duplication is unnecessary and there are already cases where
external defs are added to Value2VPValue. This patch replaces all uses
of VPExternalDefs with Value2VPValue.

It clarifies the naming of getOrAddVPValue (to getOrAddExternalVPValue)
and addVPValue (to addExternalVPValue).

At the moment, this is NFC, but will enable additional simplifications
in D147783.

Depends on D147891.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147892

show more ...

Revert "[VPlan] Unify Value2VPValue and VPExternalDefs maps (NFCI)."

Asan detects heap-use-after-free, see D147892.

This reverts commit 4fc190351e5af901b6107d162d07e1fbca90934f.
This reverts commit

Revert "[VPlan] Unify Value2VPValue and VPExternalDefs maps (NFCI)."

Asan detects heap-use-after-free, see D147892.

This reverts commit 4fc190351e5af901b6107d162d07e1fbca90934f.
This reverts commit 668045eb77628be13e448ffbb855473ffca1cc43.

show more ...

[VPlan] Unify Value2VPValue and VPExternalDefs maps (NFCI).

Before this patch, a VPlan contained 2 mappings for Values -> VPValue:
1) Value2VPValue and 2) VPExternalDefs.

This duplication is unnece

[VPlan] Unify Value2VPValue and VPExternalDefs maps (NFCI).

Before this patch, a VPlan contained 2 mappings for Values -> VPValue:
1) Value2VPValue and 2) VPExternalDefs.

This duplication is unnecessary and there are already cases where
external defs are added to Value2VPValue. This patch replaces all uses
of VPExternalDefs with Value2VPValue.

It clarifies the naming of getOrAddVPValue (to getOrAddExternalVPValue)
and addVPValue (to addExternalVPValue).

At the moment, this is NFC, but will enable additional simplifications
in D147783.

Depends on D147891.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147892

show more ...

[VPlan] Replace InvariantCond field from VPWidenSelectRecipe.

There is no need to store information about invariance in the recipe.
Replace the fields with checks of the operands using
isDefinedOuts

[VPlan] Replace InvariantCond field from VPWidenSelectRecipe.

There is no need to store information about invariance in the recipe.
Replace the fields with checks of the operands using
isDefinedOutsideVectorRegions.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D144489

show more ...

[NFC][TargetParser] Remove llvm/ADT/Triple.h

I also ran `git clang-format` to get the headers in the right order for
the new location, which has changed the order of other headers in two
files.

[VPlan] Add support for tracking UFs applicable to VPlan (NFC).

Explicitly track the UFs supported in a VPlan. This is needed to
allow transformations to restrict the UFs which are supported.

Discu

[VPlan] Add support for tracking UFs applicable to VPlan (NFC).

Explicitly track the UFs supported in a VPlan. This is needed to
allow transformations to restrict the UFs which are supported.

Discussed as separate improvement in D135017.

show more ...

[VPlan] Add field to track if intrinsic should be used for call. (NFC)

This patch moves the cost-based decision whether to use an intrinsic or
library call to the point where the recipe is created.

[VPlan] Add field to track if intrinsic should be used for call. (NFC)

This patch moves the cost-based decision whether to use an intrinsic or
library call to the point where the recipe is created. This untangles
code-gen from the cost model and also avoids doing some extra work as
the information is already computed at construction.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D132585

show more ...