[Test] Rename the test function name suffix. NFC (#114504)

[RISCV] Mark f16/bf16 lrint and llrint cost as invalid (#113924)

We currently can't lower scalable vector lrint and llrint nodes for bf16
and f16, even with zvfh, and will crash.

Mark the cost a

[RISCV] Mark f16/bf16 lrint and llrint cost as invalid (#113924)

We currently can't lower scalable vector lrint and llrint nodes for bf16
and f16, even with zvfh, and will crash.

Mark the cost as invalid for now to prevent the vectorizers from
emitting them.

Note that we can actually lower fixed-length vectors fine by scalarizing
them, but we were still undercosting these too so I've also included
them. I presume there's an opportunity to improve the codegen later on.

show more ...

[RISCV] Add cost model tests for fp rounding ops for bf16. NFC

[RISCV] Split fp rounding ops with zvfhmin nxv32f16 (#108765)

This adds zvfhmin test coverage for fceil, ffloor, fnearbyint, frint,
fround and froundeven and splits them at nxv32f16 to avoid crashi

[RISCV] Split fp rounding ops with zvfhmin nxv32f16 (#108765)

This adds zvfhmin test coverage for fceil, ffloor, fnearbyint, frint,
fround and froundeven and splits them at nxv32f16 to avoid crashing,
similarly to what we do for other nodes that we promote.

This also sets ftrunc to promote which was previously missing. We
already promote the VP version of it, vp_froundtozero.
Marking it as promoted affects some of the cost model tests since
they're no longer expanded.

show more ...

[RISCV] Add zvfhmin cost model test coverage. NFC

This adds tests coverage for zvfhmin and halfs in general in the cost
model tests.

Some existing half tests were split into separate functions so t

[RISCV] Add zvfhmin cost model test coverage. NFC

This adds tests coverage for zvfhmin and halfs in general in the cost
model tests.

Some existing half tests were split into separate functions so that if
the check prefixes diverge it won't affect the rest of the non-half
instructions.

Whilst we're here, also remove the redundant
-riscv-vector-bits-min=128 and declares.

show more ...

ISel: introduce vector ISD::LRINT, ISD::LLRINT; custom RISCV lowering (#66924)

The issue #55208 noticed that std::rint is vectorized by the
SLPVectorizer, but a very similar function, std::lrint, i

ISel: introduce vector ISD::LRINT, ISD::LLRINT; custom RISCV lowering (#66924)

The issue #55208 noticed that std::rint is vectorized by the
SLPVectorizer, but a very similar function, std::lrint, is not.
std::lrint corresponds to ISD::LRINT in the SelectionDAG, and
std::llrint is a familiar cousin corresponding to ISD::LLRINT. Now,
neither ISD::LRINT nor ISD::LLRINT have a corresponding vector variant,
and the LangRef makes this clear in the documentation of llvm.lrint.*
and llvm.llrint.*.

This patch extends the LangRef to include vector variants of
llvm.lrint.* and llvm.llrint.*, and lays the necessary ground-work of
scalarizing it for all targets. However, this patch would be devoid of
motivation unless we show the utility of these new vector variants.
Hence, the RISCV target has been chosen to implement a custom lowering
to the vfcvt.x.f.v instruction. The patch also includes a CostModel for
RISCV, and a trivial follow-up can potentially enable the SLPVectorizer
to vectorize std::lrint and std::llrint, fixing #55208.

The patch includes tests, obviously for the RISCV target, but also for
the X86, AArch64, and PowerPC targets to justify the addition of the
vector variants to the LangRef.

show more ...

[RISCV][CostModel] Add getCFInstrCost RISC-V implementation (#65599)

This patch implements getCFInstrCost TTI hook that mostly affects
LoopVectorizer decisions. It sets zero cost for PHI nodes and

[RISCV][CostModel] Add getCFInstrCost RISC-V implementation (#65599)

This patch implements getCFInstrCost TTI hook that mostly affects
LoopVectorizer decisions. It sets zero cost for PHI nodes and zero
throughput cost for branches (assuming that branches are likely to
be predicted). The implementation is similar to X86/AArch64/PowerPC
targets and reduces loop cost by excluding induction PHIs/loop latch
branches, which in turn leads to selecting smaller vectorization
factor.

show more ...

CostModel/RISCV: fix typos in fround test, vector length (#67025)

There are several typos in fround.ll, persumably caused by copy-pasting,
where there is a strange nvx5* type. From the surrounding

CostModel/RISCV: fix typos in fround test, vector length (#67025)

There are several typos in fround.ll, persumably caused by copy-pasting,
where there is a strange nvx5* type. From the surrounding code, it is
clear that this was intended to be nvx4*. Fix these typos.

show more ...

[RISCV] Customed lower vector nearbyint and rint in RISC-V.

The patch lowers vector rint/nearbyint like vp.rint/nearbyint.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org

[RISCV] Customed lower vector nearbyint and rint in RISC-V.

The patch lowers vector rint/nearbyint like vp.rint/nearbyint.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D148619

show more ...

[RISCV] Add basic cost model for vp float rounding instructions.

Reviewed By: craig.topper, reames

Differential Revision: https://reviews.llvm.org/D137766

[VP][RISCV] Add vp.nearbyint and RISC-V support.

nearbyint has the property to execute without exception.
For not modifying fflags, the patch added new machine opcode
PseudoVFROUND_NOEXCEPT_V that e

[VP][RISCV] Add vp.nearbyint and RISC-V support.

nearbyint has the property to execute without exception.
For not modifying fflags, the patch added new machine opcode
PseudoVFROUND_NOEXCEPT_V that expands vfcvt.x.f.v and vfcvt.f.x.v between a pair
of frflags and fsflags.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D137685

show more ...

[VP][RISCV] Add vp.rint and RISC-V support.

FRINT uses dynamic rounding mode instead of static rounding mode. The patch
rename VFCVT_X_F_VL to VFCVT_RM_X_F_VL for static rounding mode uses and added

[VP][RISCV] Add vp.rint and RISC-V support.

FRINT uses dynamic rounding mode instead of static rounding mode. The patch
rename VFCVT_X_F_VL to VFCVT_RM_X_F_VL for static rounding mode uses and added
new ISDNode VFCVT_X_F_VL directly selected to PseudoVFCVT_X_F_V.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D136662

show more ...

[RISCV] Inline scalar ceil/floor/trunc/rint/round/roundeven.

This avoids the call overhead as well as the the save/restore of
fflags and the snan handling in the libm function.

The save/restore of

[RISCV] Inline scalar ceil/floor/trunc/rint/round/roundeven.

This avoids the call overhead as well as the the save/restore of
fflags and the snan handling in the libm function.

The save/restore of fflags and snan handling are needed to be
correct for -ftrapping-math. I think we can ignore them in the
default environment.

The inline sequence will generate an invalid exception for nan
and an inexact exception if fractional bits are discarded.

I've used a custom inserter to explicitly create the control flow
around the float->int->float conversion.

We can probably avoid the final fsgnj after the conversion for
no signed zeros FMF, but I'll leave that for future work.

Note the comparison constant is slightly different than glibc uses.
They use 1<<53 for double, I'm using 1<<52. I believe either are valid.
Numbers >= 1<<52 can't have any fractional bits. It's ok to do the
float->int->float conversion on numbers between 1<<53 and 1<<52 since
they will all fit in 64. We only have a problem if the double can't fit
in i64

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D136508

show more ...

[RISCV] Add missing vscale x 1 cost model entries and tests.

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D136411

[RISCV] Update cost of vector roundeven to match round which uses the same sequence but a different FRM value.

Reviewed By: reames, eopXD

Differential Revision: https://reviews.llvm.org/D134978

[RISCV] Add lowering for llvm.roundeven

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D134785

[RISCV] Add cost model for vector insert/extract element.

This patch adds cost model for vector insert/extract element instructions. In RVV, we could use vector scalar move instruction to insert or

[RISCV] Add cost model for vector insert/extract element.

This patch adds cost model for vector insert/extract element instructions. In RVV, we could use vector scalar move instruction to insert or extract the first element, and use vslide to move it. But for mask vector or i64 vector in i32 target, we need special instructions to make it.

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D133007

show more ...

[RISCV] Improve vector fround lowering by changing FRM.

This is a follow up to D133238 which did this for ceil/floor.

Reviewed By: arcbbb, frasercrmck

Differential Revision: https://reviews.llvm.o

[RISCV] Improve vector fround lowering by changing FRM.

This is a follow up to D133238 which did this for ceil/floor.

Reviewed By: arcbbb, frasercrmck

Differential Revision: https://reviews.llvm.org/D133335

show more ...

[RISCV] Improve vector fceil/ffloor lowering by changing FRM.

This adds new VFCVT pseudoinstructions that take a rounding mode operand. A custom inserter is used to insert additional instructions to

[RISCV] Improve vector fceil/ffloor lowering by changing FRM.

This adds new VFCVT pseudoinstructions that take a rounding mode operand. A custom inserter is used to insert additional instructions to change FRM around the
VFCVT.

Some of this is borrowed from D122860, but takes a somewhat different direction. We may migrate to that patch, but for now I was trying to keep this as independent from
RVV intrinsics as I could.

A followup patch will use this approach for FROUND too.

Still need to fix the cost model.

Reviewed By: arcbbb

Differential Revision: https://reviews.llvm.org/D133238

show more ...

[RISCV] Correct costs for vector ceil/floor/trunc/round

Add vector costs for ceil/floor/trunc/round. As can be seen in the tests, the prior default costs were a significant under estimate of the act

[RISCV] Correct costs for vector ceil/floor/trunc/round

Add vector costs for ceil/floor/trunc/round. As can be seen in the tests, the prior default costs were a significant under estimate of the actual code generated.

These costs are computed by simply generating code with the current backend, and then counting the number of instructions. I discount one vsetvli, and ignore the return.

Differential Revision: https://reviews.llvm.org/D131967

show more ...

[CostModel] Update RUN -passes=* to double quotes to appease update scripts on windows

DOS really doesn't like `` quotes to be used in command lines

Some prep work as I'm intending to resurrect D79

[CostModel] Update RUN -passes=* to double quotes to appease update scripts on windows

DOS really doesn't like `` quotes to be used in command lines

Some prep work as I'm intending to resurrect D79483 soon

show more ...

[CostModel][RISCV] Add test coverage of floating point rounding intrinsics

These costs are fairly bogus, but at least we have baseline coverage now.