[SDAG] Add an ISD node to help lower vector.extract.last.active (#118810)

Based on feedback from the clastb codegen PR, I'm refactoring basic codegen for the vector.extract.last.active intrinsic to

[SDAG] Add an ISD node to help lower vector.extract.last.active (#118810)

Based on feedback from the clastb codegen PR, I'm refactoring basic codegen for the vector.extract.last.active intrinsic to lower to an ISD node in SelectionDAGBuilder then expand in LegalizeVectorOps, instead of doing everything in the builder.

The new ISD node (vector_find_last_active) only covers finding the index of the last active element of the mask, and extracting the element + handling passthru is left to existing ISD nodes.

show more ...

[LegalizeVectorOps][RISCV] Use VP_FP_EXTEND/ROUND when promoting VP_FP* operations. (#122784)

This preserves the original VL leading to more reuse of VL for vsetvli.
The VLOptimizer can also clean

[LegalizeVectorOps][RISCV] Use VP_FP_EXTEND/ROUND when promoting VP_FP* operations. (#122784)

This preserves the original VL leading to more reuse of VL for vsetvli.
The VLOptimizer can also clean up a lot of this, but I'm not sure if it
gets all of it.

There are some regressions in here from propagating the mask too, but
I'm not sure if that's a concern.

show more ...

[X86] Combine `uitofp <v x i32> to <v x half>` (#121809)

Closes #121793

[DAG] VectorLegalizer::ExpandUINT_TO_FLOAT- pull out repeated getValueType calls. NFC.

[X86] Support lowering of FMINIMUMNUM/FMAXIMUMNUM (#121464)

[LegalizeVectorOps] Use getBoolConstant instead of getAllOnesConstant in VectorLegalizer::UnrollVSETCC. (#121526)

This code should follow the target preference for boolean contents of a
vector type

[LegalizeVectorOps] Use getBoolConstant instead of getAllOnesConstant in VectorLegalizer::UnrollVSETCC. (#121526)

This code should follow the target preference for boolean contents of a
vector type. We shouldn't assume that true is negative one.

show more ...

[SDAG] Merge multiple-result libcall expansion into DAG.expandMultipleResultFPLibCall() (#114792)

This merges the logic for expanding both FFREXP and FSINCOS into one
method `DAG.expandMultipleResu

[SDAG] Merge multiple-result libcall expansion into DAG.expandMultipleResultFPLibCall() (#114792)

This merges the logic for expanding both FFREXP and FSINCOS into one
method `DAG.expandMultipleResultFPLibCall()`. This reduces duplication
and also allows FFREXP to benefit from the stack slot elimination
implemented for FSINCOS. This method will also be used in future to
implement more multiple-result intrinsics (such as modf and sincospi).

show more ...

[SDAG] Support expanding `FSINCOS` to vector library calls (#114039)

This shares most of its code with the scalar sincos expansion. It allows
expanding vector FSINCOS nodes to a library call from t

[SDAG] Support expanding `FSINCOS` to vector library calls (#114039)

This shares most of its code with the scalar sincos expansion. It allows
expanding vector FSINCOS nodes to a library call from the specified
`-vector-library`. The upside of this is it will mean the vectorizer
only needs to handle the sincos intrinsic, which has no memory effects,
and this can handle lowering the intrinsic to a call that takes output
pointers.

show more ...

[SDAG] Simplify `SDNodeFlags` with bitwise logic (#114061)

This patch allows using enumeration values directly and simplifies the
implementation with bitwise logic. It addresses the comment in
htt

[SDAG] Simplify `SDNodeFlags` with bitwise logic (#114061)

This patch allows using enumeration values directly and simplifies the
implementation with bitwise logic. It addresses the comment in
https://github.com/llvm/llvm-project/pull/113808#discussion_r1819923625.

show more ...

[IR] Add `llvm.sincos` intrinsic (#109825)

This adds the `llvm.sincos` intrinsic, legalization, and lowering.

The `llvm.sincos` intrinsic takes a floating-point value and returns
both the sine a

[IR] Add `llvm.sincos` intrinsic (#109825)

This adds the `llvm.sincos` intrinsic, legalization, and lowering.

The `llvm.sincos` intrinsic takes a floating-point value and returns
both the sine and cosine (as a struct).

```
declare { float, float } @llvm.sincos.f32(float %Val)
declare { double, double } @llvm.sincos.f64(double %Val)
declare { x86_fp80, x86_fp80 } @llvm.sincos.f80(x86_fp80 %Val)
declare { fp128, fp128 } @llvm.sincos.f128(fp128 %Val)
declare { ppc_fp128, ppc_fp128 } @llvm.sincos.ppcf128(ppc_fp128 %Val)
declare { <4 x float>, <4 x float> } @llvm.sincos.v4f32(<4 x float> %Val)
```

The lowering is built on top of the existing FSINCOS ISD node, with
additional type legalization to allow for f16, f128, and vector values.

show more ...

[X86][CodeGen] Add base atan2 intrinsic lowering (p4) (#110760)

This change is part of this proposal:
https://discourse.llvm.org/t/rfc-all-the-math-intrinsics/78294

Based on example PR #96222 an

[X86][CodeGen] Add base atan2 intrinsic lowering (p4) (#110760)

This change is part of this proposal:
https://discourse.llvm.org/t/rfc-all-the-math-intrinsics/78294

Based on example PR #96222 and fix PR #101268, with some differences due
to 2-arg intrinsic and intermediate refactor (RuntimeLibCalls.cpp).

- Add llvm.experimental.constrained.atan2 - Intrinsics.td,
ConstrainedOps.def, LangRef.rst
- Add to ISDOpcodes.h and TargetSelectionDAG.td, connect to intrinsic in
BasicTTIImpl.h, and LibFunc_ in SelectionDAGBuilder.cpp
- Update LegalizeDAG.cpp, LegalizeFloatTypes.cpp, LegalizeVectorOps.cpp,
and LegalizeVectorTypes.cpp
- Update isKnownNeverNaN in SelectionDAG.cpp
- Update SelectionDAGDumper.cpp
- Update libcalls - RuntimeLibcalls.def, RuntimeLibcalls.cpp
- TargetLoweringBase.cpp - Expand for vectors, promote f16
- X86ISelLowering.cpp - Expand f80, promote f32 to f64 for MSVC

Part 4 for Implement the atan2 HLSL Function #70096.

show more ...

[LLVM][CodeGen] Add lowering for scalable vector bfloat operations. (#109803)

Specifically:
fabs, fadd, fceil, fdiv, ffloor, fma, fmax, fmaxnm, fmin, fminnm,
fmul, fnearbyint, fneg, frint, fro

[LLVM][CodeGen] Add lowering for scalable vector bfloat operations. (#109803)

Specifically:
fabs, fadd, fceil, fdiv, ffloor, fma, fmax, fmaxnm, fmin, fminnm,
fmul, fnearbyint, fneg, frint, fround, froundeven, fsub, fsqrt &
ftrunc

show more ...

[LegalizeVectorOps] Enable ExpandFABS/COPYSIGN to use integer ops for fixed vectors in some cases. (#109232)

Copy the same FSUB check from ExpandFNEG to avoid breaking AArch64 and
ARM.

[LegalizeVectorOps][RISCV] Don't scalarize FNEG in ExpandFNEG if FSUB is marked Promote.

We have a special check that tries to determine if vector FP
operations are supported for the type to determi

[LegalizeVectorOps][RISCV] Don't scalarize FNEG in ExpandFNEG if FSUB is marked Promote.

We have a special check that tries to determine if vector FP
operations are supported for the type to determine whether to
scalarize or not. If FP arithmetic would be promoted, don't unroll.

This improves Zvfhmin codegen on RISC-V.

show more ...

Revert "[LegalizeVectorOps] Make the AArch64 hack in ExpandFNEG more specific."

This reverts commit 884ff9e3f9741ac282b6cf8087b8d3f62b8e138a.

Regression was reported in Halide for arm32.

[LegalizeVectorOps] Remove calls to DAG.UnrollVectorsOps from some expansion handlers. NFC (#108930)

Instead, return SDValue() to tell the caller to do the unrolling. This
is consistent with how so

[LegalizeVectorOps] Remove calls to DAG.UnrollVectorsOps from some expansion handlers. NFC (#108930)

Instead, return SDValue() to tell the caller to do the unrolling. This
is consistent with how some other handler work. Especially the handlers
that live in TLI.

ExpandBITREVERSE was rewritten to not take the Results vector an
argument.

show more ...

[LegalizeVectorOps] Make the AArch64 hack in ExpandFNEG more specific.

Only scalarize single element vectors when vector FSUB is not
supported and scalar FNEG is supported.

[LegalizeDAG][RISCV] Don't promote f16 vector ISD::FNEG/FABS/FCOPYSIGN to f32 when we don't have Zvfh. (#106652)

The fp_extend will canonicalize NaNs which is not the semantics of
FNEG/FABS/FCOPYSI

[LegalizeDAG][RISCV] Don't promote f16 vector ISD::FNEG/FABS/FCOPYSIGN to f32 when we don't have Zvfh. (#106652)

The fp_extend will canonicalize NaNs which is not the semantics of
FNEG/FABS/FCOPYSIGN.

For fixed vectors I'm scalarizing due to test changes on other targets
where the scalarization is expected. I will try to address in a follow
up.

For scalable vectors, we bitcast to integer and use integer logic ops.

show more ...

[LegalizeVectorOps] Defer UnrollVectorOp in ExpandFNEG to caller. (#106783)

Make ExpandFNEG return SDValue() when it doesn't expand. The caller
already knows how to Unroll when Results is empty.

[SDAG] Expand vector [u|s]cmp in VectorLegalizer (#106883)

Address comment
https://github.com/llvm/llvm-project/pull/106747#issuecomment-2322922855.

[LegalizeVectorOps][RISCV] Don't promote VP_FABS/FNEG/FCOPYSIGN. (#106659)

Promoting canonicalizes NaNs which changes the semantics. Bitcast to
integer and use logic ops instead.

[LegalizeVectorOps][PowerPC] Use xor to expand fneg. (#106595)

This preserves the semantis of fneg and matches what we do in
LegalizeDAG.

I kept the legal FSUB check to force unrolling for some

[LegalizeVectorOps][PowerPC] Use xor to expand fneg. (#106595)

This preserves the semantis of fneg and matches what we do in
LegalizeDAG.

I kept the legal FSUB check to force unrolling for some targets that
don't have FSUB but have XOR. On Aarch64, using xor broke some tests that
expected to see a (v1f64 (fma (insertvector_elt (f64 (fneg
(extractvectorelt X)))))) pattern.

show more ...

Scalarize the vector inputs to llvm.lround intrinsic by default. (#101054)

Verifier is updated in a different patch to let the vector types for
llvm.lround and llvm.llround intrinsics.

[LLVM] Add `llvm.experimental.vector.compress` intrinsic (#92289)

This PR adds a new vector intrinsic `@llvm.experimental.vector.compress`
to "compress" data within a vector based on a selection ma

[LLVM] Add `llvm.experimental.vector.compress` intrinsic (#92289)

This PR adds a new vector intrinsic `@llvm.experimental.vector.compress`
to "compress" data within a vector based on a selection mask, i.e., it
moves all selected values (i.e., where `mask[i] == 1`) to consecutive
lanes in the result vector. A `passthru` vector can be provided, from
which remaining lanes are filled.

The main reason for this is that the existing
`@llvm.masked.compressstore` has very strong constraints in that it can
only write values that were selected, resulting in guard branches for
all targets except AVX-512 (and even there the AMD implementation is
_very_ slow). More instruction sets support "compress" logic, but only
within registers. So to store the values, an additional store is needed.
But this combination is likely significantly faster on many target as it
avoids branches.

In follow up PRs, my plan is to add target-specific lowerings for x86,
SVE, and possibly RISCV. I also want to combine this with a store
instruction, as this is probably a common case and we can avoid some
memory writes in that case.

See [discussion in
forum](https://discourse.llvm.org/t/new-intrinsic-for-masked-vector-compress-without-store/78663)
for initial discussion on the design.

show more ...

[X86][CodeGen] Add base trig intrinsic lowerings (#96222)

This change is an implementation of
https://github.com/llvm/llvm-project/issues/87367's investigation on
supporting IEEE math operations a

[X86][CodeGen] Add base trig intrinsic lowerings (#96222)

This change is an implementation of
https://github.com/llvm/llvm-project/issues/87367's investigation on
supporting IEEE math operations as intrinsics.
Which was discussed in this RFC:
https://discourse.llvm.org/t/rfc-all-the-math-intrinsics/78294

This change adds constraint intrinsics and some lowering cases for
`acos`, `asin`, `atan`, `cosh`, `sinh`, and `tanh`.
The only x86 specific change was for f80.

https://github.com/llvm/llvm-project/issues/70079
https://github.com/llvm/llvm-project/issues/70080
https://github.com/llvm/llvm-project/issues/70081
https://github.com/llvm/llvm-project/issues/70083
https://github.com/llvm/llvm-project/issues/70084
https://github.com/llvm/llvm-project/issues/95966

The x86 lowering is going to be done in three pr changes with this being
the first.
A second PR will be put up for Loop Vectorizing and then SLPVectorizer.

The constraint intrinsics is also going to be in multiple parts, but
just 2.
This part covers just the llvm specific changes, part2 will cover clang
specifc changes and legalization for backends than have special
legalization
requirements like aarch64 and wasm.

show more ...