[RISCV] Lower fixed-length mgather/mscatter for zvfhmin/zvfbfmin (#114945)

In preparation for allowing zvfhmin and zvfbfmin in
isLegalElementTypeForRVV, this lowers fixed-length masked gathers and

[RISCV] Lower fixed-length mgather/mscatter for zvfhmin/zvfbfmin (#114945)

In preparation for allowing zvfhmin and zvfbfmin in
isLegalElementTypeForRVV, this lowers fixed-length masked gathers and
scatters

We need to mark f16 and bf16 as legal in isLegalMaskedGatherScatter
otherwise ScalarizeMaskedMemIntrin will just scalarize them, but we can
move this back into isLegalElementTypeForRVV afterwards.

The scalarized codegen required #114938, #114927 and #114915 to not
crash.

show more ...

[RISCV][TTI] Reduce cost of a build_vector pattern (#108419)

This change is actually two related changes, but they're very hard to
meaningfully separate as the second balances the first, and yet do

[RISCV][TTI] Reduce cost of a build_vector pattern (#108419)

This change is actually two related changes, but they're very hard to
meaningfully separate as the second balances the first, and yet doesn't
do much good on it's own.

First, we can reduce the cost of a build_vector pattern. Our current
costing for this defers to generic insertelement costing which isn't
unreasonable, but also isn't correct. While inserting N elements
requires N-1 slides and N vmv.s.x, doing the full build_vector only
requires N vslide1down. (Note there are other cases that our build
vector lowering can do more cheaply, this is simply the easiest upper
bound which appears to be "good enough" for SLP costing purposes.)

Second, we need to tell SLP that calls don't preserve vector registers.
Without this, SLP will vectorize scalar code which performs e.g. 4 x
float @exp calls as two <2 x float> @exp intrinsic calls. Oddly, the
costing works out that this is in fact the optimal choice - except that
we don't actually have a <2 x float> @exp, and unroll during DAG. This
would be fine (or at least cost neutral) except that the libcall for the
scalar @exp blows all vector registers. So the net effect is we added a
bunch of spills that SLP had no idea about. Thankfully, AArch64 has a
similiar problem, and has taught SLP how to reason about spill cost once
the right TTI hook is implemented.

Now, for some implications...

The SLP solution for spill costing has some inaccuracies. In particular,
it basically just guesses whether a intrinsic will be lowered to a call
or not, and can be wrong in both directions. It also has no mechanism to
differentiate on calling convention.

This has the effect of making partial vectorization (i.e. starting in
scalar) more profitable. In practice, the major effect of this is to
make it more like SLP will vectorize part of a tree in an intersecting
forrest, and then vectorize the remaining tree once those uses have been
removed.

This has the effect of biasing us slightly away from strided, or indexed
loads during vectorization - because the scalar cost is more accurately
modeled, and these instructions look relevatively less profitable.

show more ...

[RISCV] Remove -riscv-v-vector-bits-min from cost model tests. NFC

It looks like they were added to prevent fixed length vectors from
being expanded, but that's no longer the case today:
https://rev

[RISCV] Remove -riscv-v-vector-bits-min from cost model tests. NFC

It looks like they were added to prevent fixed length vectors from
being expanded, but that's no longer the case today:
https://reviews.llvm.org/D121447#3376520

show more ...

[RISCV][TTI] Model cost for insert/extract into illegal types (#106440)

We'd previously just deferred to the base implementation, but that more
or less always returns 1. This underestimates the cos

[RISCV][TTI] Model cost for insert/extract into illegal types (#106440)

We'd previously just deferred to the base implementation, but that more
or less always returns 1. This underestimates the cost of the
insert/extract, biases the SLP vectorizer towards forming illegally
typed vectors, and underestimates the cost of scalarized operations
(like unaligned scatter/gather).

show more ...

[TTI] getCommonMaskedMemoryOpCost - consistently use getScalarizationOverhead instead of ExtractElement costs for address/mask extraction. (#87771)

These aren't unknown extraction indices, we will b

[TTI] getCommonMaskedMemoryOpCost - consistently use getScalarizationOverhead instead of ExtractElement costs for address/mask extraction. (#87771)

These aren't unknown extraction indices, we will be extracting every address/mask element in sequence.

show more ...

[RISCV][CostModel] Add getCFInstrCost RISC-V implementation (#65599)

This patch implements getCFInstrCost TTI hook that mostly affects
LoopVectorizer decisions. It sets zero cost for PHI nodes and

[RISCV][CostModel] Add getCFInstrCost RISC-V implementation (#65599)

This patch implements getCFInstrCost TTI hook that mostly affects
LoopVectorizer decisions. It sets zero cost for PHI nodes and zero
throughput cost for branches (assuming that branches are likely to
be predicted). The implementation is similar to X86/AArch64/PowerPC
targets and reduces loop cost by excluding induction PHIs/loop latch
branches, which in turn leads to selecting smaller vectorization
factor.

show more ...

[RISCV] Upgrade Zvfh version to 1.0 and move out of experimental state.

This has been ratified according to https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions

Differential Revision: h

[RISCV] Upgrade Zvfh version to 1.0 and move out of experimental state.

This has been ratified according to https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions

Differential Revision: https://reviews.llvm.org/D155668

show more ...

[CostModel] Convert some tests to opaque pointers (NFC)

[RISCV] Add cost model for vector insert/extract element.

This patch adds cost model for vector insert/extract element instructions. In RVV, we could use vector scalar move instruction to insert or

[RISCV] Add cost model for vector insert/extract element.

This patch adds cost model for vector insert/extract element instructions. In RVV, we could use vector scalar move instruction to insert or extract the first element, and use vslide to move it. But for mask vector or i64 vector in i32 target, we need special instructions to make it.

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D133007

show more ...

[CostModel] Update RUN -passes=* to double quotes to appease update scripts on windows

DOS really doesn't like `` quotes to be used in command lines

Some prep work as I'm intending to resurrect D79

[CostModel] Update RUN -passes=* to double quotes to appease update scripts on windows

DOS really doesn't like `` quotes to be used in command lines

Some prep work as I'm intending to resurrect D79483 soon

show more ...

[RISCV] Add +experimental-zvfh extension to cover half types in vectors.

Currently we allow half types in vectors if the scalar Zfh extension
is enabled. This behavior is not inline with the vector

[RISCV] Add +experimental-zvfh extension to cover half types in vectors.

Currently we allow half types in vectors if the scalar Zfh extension
is enabled. This behavior is not inline with the vector spec. For f32
and f64 types, the Zve32f, Zve64f, Zve64d, and V explicitly control
the availablity of floating point types in vectors.

In order to make our compiler compliant, we either need to remove all support
for half in vectors or we need an extension to control it.

Draft spec here https://github.com/riscv/riscv-v-spec/pull/780

Reviewed By: kito-cheng

Differential Revision: https://reviews.llvm.org/D121345

show more ...

[test] Migrate '-analyze -cost-model' to '-passes=print<cost-model>'

[RISCV] Remove experimental prefix from rvv-related extensions.

Extensions affected: +v, +zve*, +zvl*

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D117860

[RISCV] update zfh and zfhmin extention to v1.0

`zfh` and `zfhmin` have been ratified, with version 1.0.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D117098

[NFC][llvm] Inclusive language: reword uses of sanity test and check

Part of continuing work to use more inclusive language. Reworded uses
of sanity check and sanity test in llvm/test/

[RISCV] Expand unaligned fixed-length vector memory accesses

RVV vectors must be aligned to their element types, so anything less is
unaligned.

For regular loads and stores, our custom-lowering of

[RISCV] Expand unaligned fixed-length vector memory accesses

RVV vectors must be aligned to their element types, so anything less is
unaligned.

For regular loads and stores, our custom-lowering of fixed-length
vectors meant that we opted out of LegalizeDAG's built-in unaligned
expansion. This patch adds that logic in to our custom lower function.

For masked intrinsics, we declare that anything unaligned is not legal,
leaving the ScalarizeMaskedMemIntrin pass to do the expansion for us.

Note that neither of these methods can handle the expansion of
scalable-vector memory ops, so those cases are left alone by this patch.
Scalable loads and stores already go through expansion by default but
hit an assertion, and scalable masked intrinsics will silently generate
incorrect code. It may be prudent to return an error in both of these
cases.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D102493

show more ...

[RISCV][NFC] Correct alignment in scatter/gather tests

This lays the groundwork for changes to alignment in D102493 to be more
apparent.

[RISCV] Reorder checks in RISCVTTIImpl::getGatherScatterOpCost to avoid calling getMinRVVVectorSizeInBits() when V extension is not enabled.

getMinRVVVectorSizeInBits() asserts if the V extension is

[RISCV] Reorder checks in RISCVTTIImpl::getGatherScatterOpCost to avoid calling getMinRVVVectorSizeInBits() when V extension is not enabled.

getMinRVVVectorSizeInBits() asserts if the V extension isn't
enabled. So check that gather/scatter is legal first since it
already contains a check for V extension being enabled. It
also already checks getMinRVVVectorSizeInBits for fixed length
vectors so we don't need a check in getGatherScatterOpCost.

show more ...

[RISCV] Add basic cost modelling for fixed vector gather/scatter.

Reviewed By: frasercrmck

Differential Revision: https://reviews.llvm.org/D99142