[RISCV] Lower fixed-length mgather/mscatter for zvfhmin/zvfbfmin (#114945)

In preparation for allowing zvfhmin and zvfbfmin in
isLegalElementTypeForRVV, this lowers fixed-length masked gathers and

[RISCV] Lower fixed-length mgather/mscatter for zvfhmin/zvfbfmin (#114945)

In preparation for allowing zvfhmin and zvfbfmin in
isLegalElementTypeForRVV, this lowers fixed-length masked gathers and
scatters

We need to mark f16 and bf16 as legal in isLegalMaskedGatherScatter
otherwise ScalarizeMaskedMemIntrin will just scalarize them, but we can
move this back into isLegalElementTypeForRVV afterwards.

The scalarized codegen required #114938, #114927 and #114915 to not
crash.

show more ...

[RISCV] Add zvfhmin cost model test coverage. NFC

This adds tests coverage for zvfhmin and halfs in general in the cost
model tests.

Some existing half tests were split into separate functions so t

[RISCV] Add zvfhmin cost model test coverage. NFC

This adds tests coverage for zvfhmin and halfs in general in the cost
model tests.

Some existing half tests were split into separate functions so that if
the check prefixes diverge it won't affect the rest of the non-half
instructions.

Whilst we're here, also remove the redundant
-riscv-vector-bits-min=128 and declares.

show more ...

[RISCV][TTI] Add legality check of vector of address for gather/scatter. (#106481)

This patch add a legality check that checks if target machine support
vector of address in `isLegalMaskedGatherSca

[RISCV][TTI] Add legality check of vector of address for gather/scatter. (#106481)

This patch add a legality check that checks if target machine support
vector of address in `isLegalMaskedGatherScatter()`.

show more ...

[RISCV][CostModel] Add getCFInstrCost RISC-V implementation (#65599)

This patch implements getCFInstrCost TTI hook that mostly affects
LoopVectorizer decisions. It sets zero cost for PHI nodes and

[RISCV][CostModel] Add getCFInstrCost RISC-V implementation (#65599)

This patch implements getCFInstrCost TTI hook that mostly affects
LoopVectorizer decisions. It sets zero cost for PHI nodes and zero
throughput cost for branches (assuming that branches are likely to
be predicted). The implementation is similar to X86/AArch64/PowerPC
targets and reduces loop cost by excluding induction PHIs/loop latch
branches, which in turn leads to selecting smaller vectorization
factor.

show more ...

[RISCV] Upgrade Zvfh version to 1.0 and move out of experimental state.

This has been ratified according to https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions

Differential Revision: h

[RISCV] Upgrade Zvfh version to 1.0 and move out of experimental state.

This has been ratified according to https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions

Differential Revision: https://reviews.llvm.org/D155668

show more ...

[CostModel] Convert some tests to opaque pointers (NFC)

[RISCV] Use VScaleForTuning in costing of operations whose cost depends on VL

On known hardware, reductions, gather, and scatter operations have execution latencies which correlated with the vector

[RISCV] Use VScaleForTuning in costing of operations whose cost depends on VL

On known hardware, reductions, gather, and scatter operations have execution latencies which correlated with the vector length (VL) of the operation. Most other operations (e.g. simply arithmetic) don't correlated in this way, and instead essentially fixed cost as VL varies.

When I'd implemented initial scalable cost model support for reductions, gather, and scatter operations, I had used an upper bound on the statically unknown VL. The argument at the time was that this prevented falsely low costs, and biased the vectorizer away from generating bad (on some hardware) code. Unfortunately, practical experience shows we were a bit too effective at that goal, and the high costs defacto prevents vectorization using these constructs at all.

This patch reverses course, and ties the returned cost not to the maximum possible VL, but the VL which would correspond to VScaleForTuning. This parameter is the same one the vectorizer uses when normalizing loop costs, so the term effectively cancels out. The result is that the vectorizer now sees these constructs as comparable in cost to their fixed length variants.

This does introduce the possibility of the cost for these operations being a significant under estimate on platforms where actual VLEN is far from that implied by VScaleForTuning. On such platforms, we might make poor heuristic choices. Probably not in LV itself (due to the cancellation mentioned above), but possibly during e.g. lowering. I'm not currently aware of any concrete examples of this, but this patch does open a concern which did not previously exist.

Previously, we had the problem of overestimating costs causing the same problem on machines much closer to default values for vscale for tuning. With this patch, we still have that problem potentially if vscale for tuning is set high (manually), and then the code is run on a narrow VLEN machine.

Differential Revision: https://reviews.llvm.org/D131519

show more ...

[CostModel] Update RUN -passes=* to double quotes to appease update scripts on windows

DOS really doesn't like `` quotes to be used in command lines

Some prep work as I'm intending to resurrect D79

[CostModel] Update RUN -passes=* to double quotes to appease update scripts on windows

DOS really doesn't like `` quotes to be used in command lines

Some prep work as I'm intending to resurrect D79483 soon

show more ...

[RISCV] Fix crash when costing scalable gather/scatter of pointer

This was a bug introduced in d764aa. A pointer type is not a primitive type, and thus we were ending up dividing by zero when comput

[RISCV] Fix crash when costing scalable gather/scatter of pointer

This was a bug introduced in d764aa. A pointer type is not a primitive type, and thus we were ending up dividing by zero when computing VLMax.

Differential Revision: https://reviews.llvm.org/D128219

show more ...

[RISCV] Add cost model for scalable scatter and gather

The costing we use for fixed length vector gather and scatter is to simply count up the memory ops, and multiply by a fixed memory op cost. For

[RISCV] Add cost model for scalable scatter and gather

The costing we use for fixed length vector gather and scatter is to simply count up the memory ops, and multiply by a fixed memory op cost. For scalable vectors, we don't actually know how many lanes are active. Instead, we have to end up making a worst case assumption on how many lanes could be active. In the generic +V case, this results in very high costs, but we can do better when we know an upper bound on the VLEN.

There's some obvious ways to improve this - e.g. using information about VL and mask bits from the instruction to reduce the upper bound - but this seems like a reasonable starting point.

The resulting costs do bias us pretty strongly away from generating scatter/gather for generic +V. Without this, we'd be returning an invalid cost and thus definitely not vectorizing, so no major change in practical behavior expected.

Differential Revision: https://reviews.llvm.org/D127541

show more ...

[RISCV] Minor test improvements for scalable scatter/gather tests added in 275b2e524

[RISCV] Add cost model coverage for scalable scatter/gather