[VPlan] Verify scalar types in VPlanVerifier. NFCI (#122679)

VTypeAnalysis contains some assertions which can be useful for reasoning
that the types of various operands match.

This patch teaches VP

[VPlan] Verify scalar types in VPlanVerifier. NFCI (#122679)

VTypeAnalysis contains some assertions which can be useful for reasoning
that the types of various operands match.

This patch teaches VPlanVerifier to invoke VTypeAnalysis to check them,
and catches some issues with VPInstruction types that are also fixed
here:

* Handles the missing cases for CalculateTripCountMinusVF,
CanonicalIVIncrementForPart and AnyOf
* Fixes ICmp and ActiveLaneMask to return i1 (to align with `icmp` and
`@llvm.get.active.lane.mask` in the LangRef)

The VPlanVerifier unit tests also need to be fleshed out a bit more to
satisfy the stricter assertions

show more ...

[VPlan] Manage created blocks directly in VPlan. (NFC) (#120918)

This patch changes the way blocks are managed by VPlan. Previously all
blocks reachable from entry would be cleaned up when a VPlan

[VPlan] Manage created blocks directly in VPlan. (NFC) (#120918)

This patch changes the way blocks are managed by VPlan. Previously all
blocks reachable from entry would be cleaned up when a VPlan is
destroyed. With this patch, each VPlan keeps track of blocks created for
it in a list and this list is then used to delete all blocks in the list
when the VPlan is destroyed. To do so, block creation is funneled
through helpers in directly in VPlan.

The main advantage of doing so is it simplifies CFG transformations, as
those do not have to take care of deleting any blocks, just adjusting
the CFG. This helps to simplify
https://github.com/llvm/llvm-project/pull/108378 and
https://github.com/llvm/llvm-project/pull/106748.

This also simplifies handling of 'immutable' blocks a VPlan holds
references to, which at the moment only include the scalar header block.

PR: https://github.com/llvm/llvm-project/pull/120918

show more ...

[VPlan] Always create initial blocks in constructor (NFC).

Update C++ unit tests to use VPlanTestBase to construct initial VPlan,
using a constructor that creates the VP blocks directly in the
const

[VPlan] Always create initial blocks in constructor (NFC).

Update C++ unit tests to use VPlanTestBase to construct initial VPlan,
using a constructor that creates the VP blocks directly in the
constructor.

Split off from and in preparation for
https://github.com/llvm/llvm-project/pull/120918.

show more ...

[VPlan] Don't use VPlan ctor taking trip count in most unit tests (NFC).

Update tests to use constructor not passing a trip count VPValue. The
tests don't need that and are simpler as a result.

[VPlan] Remove legacy VPlan() constructors (NFC).

The constructors were retained to reduce the diff during transition.

Remove them now.

[VPlan] Hook IR blocks into VPlan during skeleton creation (NFC) (#114292)

As a first step to move towards modeling the full skeleton in VPlan,
start by wrapping IR blocks created during legacy ske

[VPlan] Hook IR blocks into VPlan during skeleton creation (NFC) (#114292)

As a first step to move towards modeling the full skeleton in VPlan,
start by wrapping IR blocks created during legacy skeleton creation in
VPIRBasicBlocks and hook them into the VPlan. This means the skeleton
CFG is represented in VPlan, just before execute. This allows moving
parts of skeleton creation into recipes in the VPBBs gradually.

Note that this allows retiring some manual DT updates, as this will be
handled automatically during VPlan execution.

PR: https://github.com/llvm/llvm-project/pull/114292

show more ...

[VPlan] Don't leak ScalarHeader BasicBlock in unit tests.

[VPlan] Connect scalar header to VPlan CFG in unit tests.

This ensures the VPIRBasicBlocks are deleted when the VPlan is
destroyed.

Fixes a buildbot failure with ASAN, including
https://lab.llvm.or

[VPlan] Connect scalar header to VPlan CFG in unit tests.

This ensures the VPIRBasicBlocks are deleted when the VPlan is
destroyed.

Fixes a buildbot failure with ASAN, including
https://lab.llvm.org/buildbot/#/builders/52/builds/3368

show more ...

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handl

[VPlan] Introduce scalar loop header in plan, remove VPLiveOut. (#109975)

Update VPlan to include the scalar loop header. This allows retiring
VPLiveOut, as the remaining live-outs can now be handled by adding
operands to the wrapped phis in the scalar loop header.

Note that the current version only includes the scalar loop header, no
other loop blocks and also does not wrap it in a region block.

PR: https://github.com/llvm/llvm-project/pull/109975

show more ...

[VPlan] Merge main VPlan verifer with HCFG verifier.

Unify VPlan verifiers in verifyVPlanIsValid. This adds verification for
various properties on blocks to the verifier used for VPlans generated
by

[VPlan] Merge main VPlan verifer with HCFG verifier.

Unify VPlan verifiers in verifyVPlanIsValid. This adds verification for
various properties on blocks to the verifier used for VPlans generated
by the inner loop vectorizer. It also adds def-use checks for the
verifier used in the VPlan native path.

This drops the separate flag to enable HCFG verification. Instead, all
VPlans are verified once they have been created, if assertions are
enabled.

This also removes VPWidenPHIRecipe from VPHeaderPHIRecipe; it is used to
model any phi node in the native path.

show more ...

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is

[VPlan] Introduce new entry block to VPlan for early SCEV expansion.

This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is disconnected
at the moment, as the bypass blocks of the skeleton are not yet modeled
in VPlan.

The preheader block is executed before skeleton creation, so the SCEV
expansion results can be used during skeleton creation. At the moment,
the trip count expression and induction steps are expanded in the new
preheader. The remainder of SCEV expansions will be moved gradually in
the future.

D147965 will update skeleton creation to use the steps expanded in the
pre-header to fix #58811.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D147964

show more ...

[VPlan] Use VPDominatorTree in VPlanVerifier .

Use VPDominatorTree to generalize def-use verification.

Depends on D140513.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D140514

[VPlan] Fix leak by manually cleaning up allocated Phi in test.

This should fix a LeakSanitizer failure reported here:
https://lab.llvm.org/buildbot/#/builders/5/builds/30952

[VPlan] Add tests for VPlanVerifier (NFC).

Extra test coverage suggested for D140514.