[RISCV] Clear vill for whole vector register moves in vsetvli insertion (#118283)

This is an alternative to #117866 that works by demanding a valid vtype
instead of using a separate pass.

The ma

[RISCV] Clear vill for whole vector register moves in vsetvli insertion (#118283)

This is an alternative to #117866 that works by demanding a valid vtype
instead of using a separate pass.

The main advantage of this is that it allows coalesceVSETVLIs to just
reuse an existing vsetvli later in the block.

To do this we need to first transfer the vsetvli info to some arbitrary
valid state in transferBefore when we encounter a vector copy. Then we
add a new vill demanded field that will happily accept any other known
vtype, which allows us to coalesce these where possible.

Note we also need to check for vector copies in computeVLVTYPEChanges,
otherwise the pass will completely skip over functions that only have
vector copies and nothing else.

This is one part of a fix for #114518. We still need to check if there's
other cases where vector copies/whole register moves that are inserted
after vsetvli insertion.

show more ...

[RISCV] Enable bidirectional scheduling and tracking register pressure (#115445)


This is based on other targets like PPC/AArch64 and some experiments.

This PR will only enable bidirectional schedu

[RISCV] Enable bidirectional scheduling and tracking register pressure (#115445)


This is based on other targets like PPC/AArch64 and some experiments.

This PR will only enable bidirectional scheduling and tracking register
pressure.

Disclaimer: I haven't tested it on many cores, maybe we should make
some options being features. I believe downstreams must have tried
this before, so feedbacks are welcome.

show more ...

[RISCV] Remove redundant +zfh from +zvfh[min] tests. NFC

In the vast majority of f16 tests we don't end up emitting any scalar
code that needs +zfh, so remove it.

[RISCV] Remove experimental for bf16 extensions (#97996)

They are already ratified now.

[RISCV] Strength reduce mul by 2^N - 2^M (#88983)

This is a three instruction expansion, and does not depend on zba, so
most of the test changes are in base RV32/64I configurations.

With zba, th

[RISCV] Strength reduce mul by 2^N - 2^M (#88983)

This is a three instruction expansion, and does not depend on zba, so
most of the test changes are in base RV32/64I configurations.

With zba, this gets immediates such as 14, 28, 30, 56, 60, 62.. which
aren't covered by our other expansions.

show more ...

[RISCV] Add codegen support for Zvfbfmin (#87911)

This patch adds basic codegen support for Zvfbfmin extension.

[RISCV] Recursively split concat_vector into smaller LMULs when lowering (#85825)

This is a reimplementation of the combine added in #83035 but as a
lowering instead of a combine, so we don't regre

[RISCV] Recursively split concat_vector into smaller LMULs when lowering (#85825)

This is a reimplementation of the combine added in #83035 but as a
lowering instead of a combine, so we don't regress the test case added
in e59f120e3a14ccdc55fcb7be996efaa768daabe0 by interfering with the
strided load combine

Previously the combine had to concatenate the split vectors with
insert_subvector instead of concat_vectors to prevent an infinite
combine loop. And the reasoning behind keeping it as a combine was
because if we emitted the insert_subvector during lowering then we
didn't fold away inserts of undef subvectors.

However it turns out we can avoid this if we just do this in lowering
and select a concat_vector directly, since we get the undef folding for
free with `DAG.getNode(ISD::CONCAT_VECTOR, ...)` via foldCONCAT_VECTORS.

show more ...

Revert "[RISCV] Recursively split concat_vector into smaller LMULs (#83035)"

This reverts commit c59129a7c79448837d665de8f2743ad4b14666f6.

This causes regressions in some x264 workloads like pixel_

Revert "[RISCV] Recursively split concat_vector into smaller LMULs (#83035)"

This reverts commit c59129a7c79448837d665de8f2743ad4b14666f6.

This causes regressions in some x264 workloads like pixel_var_8x8 due to it
interfering with the strided load combine. Reverting so I can try to rework
it as a lowering instead.

show more ...

[RISCV] Recursively split concat_vector into smaller LMULs (#83035)

This is the concat_vector equivalent of #81312, in that we recursively
split concat_vectors with more than two operands into smal

[RISCV] Recursively split concat_vector into smaller LMULs (#83035)

This is the concat_vector equivalent of #81312, in that we recursively
split concat_vectors with more than two operands into smaller
concat_vectors.

This allows us to break up the chain of vslideups, as well as perform
the vslideups at a smaller LMUL, which in turn reduces register pressure
as the previous lowering performed N vslideups at the highest result
LMUL. For now, it stops splitting past MF2.

This is done as a DAG combine so that any undef operands are combined
away: If we do this during lowering then we end up with unnecessary
vslideups of undefs.

show more ...

[RISCV] Use a ta vslideup if inserting over end of InterSubVT (#83230)

The description in #83146 is slightly inaccurate: it relaxes a tail
undisturbed vslideup to tail agnostic if we are inserting

[RISCV] Use a ta vslideup if inserting over end of InterSubVT (#83230)

The description in #83146 is slightly inaccurate: it relaxes a tail
undisturbed vslideup to tail agnostic if we are inserting over the
entire tail of the vector **and** we didn't shrink the LMUL of the
vector being inserted into.

This handles the case where we did shrink down the LMUL via InterSubVT
by checking if we inserted over the entire tail of InterSubVT, the
actual type that we're performing the vslideup on, not VecVT.

show more ...

[DAG] Fold insert_subvector undef, (extract_subvector X, 0), 0 with non-matching types

We have an existing DAG combine for when an insert/extract subvector pair is entirely a nop, but we hadn't hand

[DAG] Fold insert_subvector undef, (extract_subvector X, 0), 0 with non-matching types

We have an existing DAG combine for when an insert/extract subvector pair is entirely a nop, but we hadn't handled the case where the net result was either an insert or an extract (but not both). The transform is restricted to index = 0 to avoid having to adjust indices after the transform.

Differential Revision: https://reviews.llvm.org/D158201

show more ...

Revert "[DAG] Fold insert_subvector undef, (extract_subvector X, 0), 0 with non-matching types"

This reverts commit 770be43f6782dab84d215d01b37396d63a9c2b6e.

Forgot to remove from my tree while exp

Revert "[DAG] Fold insert_subvector undef, (extract_subvector X, 0), 0 with non-matching types"

This reverts commit 770be43f6782dab84d215d01b37396d63a9c2b6e.

Forgot to remove from my tree while experimenting.

show more ...

[DAG] Fold insert_subvector undef, (extract_subvector X, 0), 0 with non-matching types

We have an existing DAG combine for when an insert/extract subvector pair is entirely a nop, but we hadn't hand

[DAG] Fold insert_subvector undef, (extract_subvector X, 0), 0 with non-matching types

We have an existing DAG combine for when an insert/extract subvector pair is entirely a nop, but we hadn't handled the case where the net result was either an insert or an extract (but not both). The transform is restricted to index = 0 to avoid having to adjust indices after the transform.

Reviews, a couple comments on the test changes:
* Mostly RISCV, mostly schedule reordering.
* One real regression in splats-with-mixed-vl.ll due to a different overly aggressive combine, fix in a follow up patch.
* The test/CodeGen/X86/vector-replicaton-i1-mask.ll diff looked concerning at first, but not the mask size at most 4 i1s. I think the type changes on the mask loads are correct, but would welcome a second opinion with someone more familiar with AVX512 codegen.

Differential Revision: https://reviews.llvm.org/D158201

show more ...

[RISCV] Upgrade Zvfh version to 1.0 and move out of experimental state.

This has been ratified according to https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions

Differential Revision: h

[RISCV] Upgrade Zvfh version to 1.0 and move out of experimental state.

This has been ratified according to https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions

Differential Revision: https://reviews.llvm.org/D155668

show more ...

[RISCV] Fold ops into vmv.v.v as vmerge with all-ones mask

A vmv.v.v shares the same encoding as a vmerge that isn't masked, so we can
also fold it into its operands if we treat it as a vmerge with

[RISCV] Fold ops into vmv.v.v as vmerge with all-ones mask

A vmv.v.v shares the same encoding as a vmerge that isn't masked, so we can
also fold it into its operands if we treat it as a vmerge with an all-ones
mask. We take care here not to actually transform the existing vmv into a
vmerge, otherwise things like True.hasOneUse() become inaccurate. Instead this
just returns an equivalent list of operands.
This is an alternative to D153351.

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D155101

show more ...

[RISCV] Use tail undisturbed vmv.v.v instead of vslideup.vi vN, vM, 0 for subvector insertion

vslideup has a vector overlap constraint that vmv.v.v doesn't.
vmv.v.v is also a simpler instruction so

[RISCV] Use tail undisturbed vmv.v.v instead of vslideup.vi vN, vM, 0 for subvector insertion

vslideup has a vector overlap constraint that vmv.v.v doesn't.
vmv.v.v is also a simpler instruction so may have better
throughput and/or latency in some CPUs.

This is an alternative to D152298, D152368, and D152496.

Reviewed By: luke, reames

Differential Revision: https://reviews.llvm.org/D152565

show more ...

[RISCV] Enable subregister liveness by default

This commit enable the subregister liveness by default in RISC-V.

It was previously disabled in https://reviews.llvm.org/D129646 after a previous atte

[RISCV] Enable subregister liveness by default

This commit enable the subregister liveness by default in RISC-V.

It was previously disabled in https://reviews.llvm.org/D129646 after a previous attempt to enabled it https://reviews.llvm.org/D128016.

We believe that https://reviews.llvm.org/D129735 fixes the issue that caused it to be disabled.

Reviewed By: craig.topper, kito-cheng

Differential Revision: https://reviews.llvm.org/D145546

show more ...

[RISCV][InsertVSETVLI] Default to MA not MU

This changes the default value used for mask policy from mask undisturbed to mask agnostic. In hardware, there may be a minor preference for ta/ma, but si

[RISCV][InsertVSETVLI] Default to MA not MU

This changes the default value used for mask policy from mask undisturbed to mask agnostic. In hardware, there may be a minor preference for ta/ma, but since this is only going to apply to instructions which don't use the mask policy bit, this is functionally mostly a nop. The main value is to make future changes to using MA when legal for masked instructions easier to review by reducing test churn.

The prior code was motivated by a desire to minimize state transitions between masked and unmasked code. This patch achieves the same effect using the demanded field logic (landed in afb45ff), and there are no regressions I spotted in the test diffs. (Given the size, I have only been able to skim.) I do want to call out that regressions are possible here; the demanded analysis only works on a block local scope right now, so e.g. a tight loop mixing masked and unmasked computation might see an extra vsetvli or two.

Differential Revision: https://reviews.llvm.org/D133803

show more ...

[RISCV] Disable subregister liveness by default

We previously enabled subregister liveness by default when compiling
with RVV. This has been shown to cause miscompilations where RVV
register operand

[RISCV] Disable subregister liveness by default

We previously enabled subregister liveness by default when compiling
with RVV. This has been shown to cause miscompilations where RVV
register operand constraints are not met. A test was added for this in
D129639 which explains the issue in more detail.

Until this issue is fixed in some way, we should not be enabling
subregister liveness unless the user asks for it.

Reviewed By: craig.topper, rogfer01, kito-cheng

Differential Revision: https://reviews.llvm.org/D129646

show more ...

[IR] Move vector.insert/vector.extract out of experimental namespace

These intrinsics are now fundemental for SVE code generation and have been
present for a year and a half, hence move them out of

[IR] Move vector.insert/vector.extract out of experimental namespace

These intrinsics are now fundemental for SVE code generation and have been
present for a year and a half, hence move them out of the experimental
namespace.

Differential Revision: https://reviews.llvm.org/D127976

show more ...

Recommit "[RISCV] Enable subregister liveness tracking for RVV."

The failure that caused the previous revert has been fixed
by https://reviews.llvm.org/D126048

Original commit message:

RVV makes h

Recommit "[RISCV] Enable subregister liveness tracking for RVV."

The failure that caused the previous revert has been fixed
by https://reviews.llvm.org/D126048

Original commit message:

RVV makes heavy use of subregisters due to LMUL>1 and segment
load/store tuples. Enabling subregister liveness tracking improves the quality
of the register allocation.

I've added a command line that can be used to turn it off if it causes compile
time or functional issues. I used the command line to keep the old behavior
for one interesting test case that was testing register allocation.

Reviewed By: kito-cheng

Differential Revision: https://reviews.llvm.org/D128016

show more ...

[DAG] Fold (shl (srl x, c), c) -> and(x, m) even if srl has other uses

If we're using shift pairs to mask, then relax the one use limit if the shift amounts are equal - we'll only be generating a si

[DAG] Fold (shl (srl x, c), c) -> and(x, m) even if srl has other uses

If we're using shift pairs to mask, then relax the one use limit if the shift amounts are equal - we'll only be generating a single AND node.

AArch64 has a couple of regressions due to this, so I've enforced the existing one use limit inside a AArch64TargetLowering::shouldFoldConstantShiftPairToMask callback.

Part of the work to fix the regressions in D77804

Differential Revision: https://reviews.llvm.org/D125607

show more ...

[RISCV] Fix incorrect use of tail agnostic vslideup.

We need to use tail undisturbed for vslideup to implement
vector insert operation correctly.

Ideally, we cound use the tail agnostic when insert

[RISCV] Fix incorrect use of tail agnostic vslideup.

We need to use tail undisturbed for vslideup to implement
vector insert operation correctly.

Ideally, we cound use the tail agnostic when insert subvector
or element at the end of the vector. This will be in follow-up
patch.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D125545

show more ...

Revert "[RISCV] Enable subregister liveness tracking for RVV."

This reverts most of ed242b54c9c2aa84a47f66af5b8497d93646b68d

I'm seeing failures in our intrinsic testing on qemu that seem
related t

Revert "[RISCV] Enable subregister liveness tracking for RVV."

This reverts most of ed242b54c9c2aa84a47f66af5b8497d93646b68d

I'm seeing failures in our intrinsic testing on qemu that seem
related to this. Reverting while I investigate.

I've left the command line option in place for directed testing.
It defaults to off.

show more ...

[RISCV] Enable subregister liveness tracking for RVV.

RVV makes heavy use of subregisters due to LMUL>1 and segment
load/store tuples. Enabling subregister liveness tracking improves the quality
of

[RISCV] Enable subregister liveness tracking for RVV.

RVV makes heavy use of subregisters due to LMUL>1 and segment
load/store tuples. Enabling subregister liveness tracking improves the quality
of the register allocation.

I've added a command line that can be used to turn it off if it causes compile
time or functional issues. I used the command line to keep the old behavior
for one interesting test case that was testing register allocation.

Reviewed By: frasercrmck

Differential Revision: https://reviews.llvm.org/D125108

show more ...