[TargetLowering] Inline the only caller of one of the forceExpandWideMUL functions. NFC

This caller does not need the libcall portion so it can directly
call forceExpandMultiply.

[SDAG] Add an ISD node to help lower vector.extract.last.active (#118810)

Based on feedback from the clastb codegen PR, I'm refactoring basic codegen for the vector.extract.last.active intrinsic to

[SDAG] Add an ISD node to help lower vector.extract.last.active (#118810)

Based on feedback from the clastb codegen PR, I'm refactoring basic codegen for the vector.extract.last.active intrinsic to lower to an ISD node in SelectionDAGBuilder then expand in LegalizeVectorOps, instead of doing everything in the builder.

The new ISD node (vector_find_last_active) only covers finding the index of the last active element of the mask, and extracting the element + handling passthru is left to existing ISD nodes.

show more ...

[LegalizeIntegerTypes] Use forceExpandWideMUL in ExpandIntRes_XMULO. (#123432)

This generates basically the same code with the operands commuted, but
gets there with less legalization steps.

[llvm][CodeGen] Intrinsic `llvm.powi.*` code gen for vector arguments (#118242)

Scalarize vector FPOWI instead of promoting the type. This allows the
scalar FPOWIs to be visited and converted to li

[llvm][CodeGen] Intrinsic `llvm.powi.*` code gen for vector arguments (#118242)

Scalarize vector FPOWI instead of promoting the type. This allows the
scalar FPOWIs to be visited and converted to libcalls before promoting
the type.

FIXME: This should be done in LegalizeVectorOps/LegalizeDAG, but call
lowering needs the unpromoted EVT.

Without this patch, in some backends, such as RISCV64 and LoongArch64,
the i32 type is illegal and will be promoted. This causes exponent type
check to fail when ISD::FPOWI node generates a libcall.

Fix https://github.com/llvm/llvm-project/issues/118079

show more ...

Reapply "[DAGCombiner] Add support for scalarising extracts of a vector setcc (#117566)" (#118823)

[Reverts d57892a2a153ab71a796f07e39d939eae6910c21]

For IR like this:

%icmp = icmp ult <4 x i

Reapply "[DAGCombiner] Add support for scalarising extracts of a vector setcc (#117566)" (#118823)

[Reverts d57892a2a153ab71a796f07e39d939eae6910c21]

For IR like this:

%icmp = icmp ult <4 x i32> %a, splat (i32 5)
%res = extractelement <4 x i1> %icmp, i32 1

where there is only one use of %icmp we can take a similar approach
to what we already do for binary ops such add, sub, etc. and convert
this into

%ext = extractelement <4 x i32> %a, i32 1
%res = icmp ult i32 %ext, 5

For AArch64 targets at least the scalar boolean result will almost
certainly need to be in a GPR anyway, since it will probably be
used by branches for control flow. I've tried to reuse existing code
in scalarizeExtractedBinop to also work for setcc.

NOTE: The optimisations don't apply for tests such as
extract_icmp_v4i32_splat_rhs in the file

CodeGen/AArch64/extract-vector-cmp.ll

because scalarizeExtractedBinOp only works if one of the input
operands is a constant.

---------

Co-authored-by: Paul Walker <paul.walker@arm.com>

show more ...

[SelectionDAG] Rename CallOptions::IsSExt to IsSigned. NFC (#118574)

This is eventually passed to shouldSignExtendTypeInLibCall which calls
it IsSigned.

[LegalizeTypes][RISCV] Call setTypeListBeforeSoften from ExpandIntRes_FP_TO_XINT if the FP type needs to be softened (#118269)

This avoids an unnecessary sext.w before the libcall.

[SelectionDAG][RISCV][AArch64] Allow f16 STRICT_FLDEXP to be promoted. Fix integer promotion of STRICT_FLDEXP in type legalizer. (#117633)

A special case in type legalization wasn't accounting for d

[SelectionDAG][RISCV][AArch64] Allow f16 STRICT_FLDEXP to be promoted. Fix integer promotion of STRICT_FLDEXP in type legalizer. (#117633)

A special case in type legalization wasn't accounting for different
operand numbering between FLDEXP and STRICT_FLDEXP.

AArch64 already asked STRICT_FLDEXP to be promoted, but had no test for
it.

show more ...

[SelectionDAG] Support integer promotion for VP_LOAD and VP_STORE (#81299)

Add integer promotion support for for VP_LOAD and VP_STORE via legalization of extend
and truncate of each form.

Patch

[SelectionDAG] Support integer promotion for VP_LOAD and VP_STORE (#81299)

Add integer promotion support for for VP_LOAD and VP_STORE via legalization of extend
and truncate of each form.

Patch commandeered from: https://reviews.llvm.org/D109377

show more ...

[SDAG] Simplify `SDNodeFlags` with bitwise logic (#114061)

This patch allows using enumeration values directly and simplifies the
implementation with bitwise logic. It addresses the comment in
htt

[SDAG] Simplify `SDNodeFlags` with bitwise logic (#114061)

This patch allows using enumeration values directly and simplifies the
implementation with bitwise logic. It addresses the comment in
https://github.com/llvm/llvm-project/pull/113808#discussion_r1819923625.

show more ...

[Codegen][LegalizeIntegerTypes] Improve shift through stack (#96151)

Minor improvement on cc39c3b17fb2598e20ca0854f9fe6d69169d85c7.

Use an aligned stack slot to store the shifted value.
Use the

[Codegen][LegalizeIntegerTypes] Improve shift through stack (#96151)

Minor improvement on cc39c3b17fb2598e20ca0854f9fe6d69169d85c7.

Use an aligned stack slot to store the shifted value.
Use the native register width as shifting unit, so the load of the
shift result is aligned.

If the shift amount is a multiple of the native register width, there is
no need to do a follow-up shift after the load. I added new tests for
these cases.

Co-authored-by: Gergely Futo <gergely.futo@hightec-rt.com>

show more ...

[LegalizeIntegerTypes] Simplify ExpandIntRes_FP_TO_XINT when operand needs to be SoftPromoted. (#107634)

Create an FP_EXTEND instead of handling the soft promote directly. This
FP_EXTEND will be vi

[LegalizeIntegerTypes] Simplify ExpandIntRes_FP_TO_XINT when operand needs to be SoftPromoted. (#107634)

Create an FP_EXTEND instead of handling the soft promote directly. This
FP_EXTEND will be visited and soft promoted itself.

This removes a zero extend from the generated code when the f32 type is
itself softened. Previously we softened it as an fp16_to_fp which sees
the operand as an integer type so we extend it. When we soften the
result as an fp_extend we see the source as f16 and don't extend. It
only becomes an integer inside call lowering not by type legalization.

If this extend is really necessary, then we have an issue when an
f16->f32 fp_extend exists in the source and f32 needs to be softened.

This simplifies part of #102503.

show more ...

[LegalizeTypes] Avoid creating an unused node in ExpandIntRes_ADDSUB. NFC

The Hi result is sometimes calculated a different way and this
node goes unused. Defer creation until we know for sure it is

[LegalizeTypes] Avoid creating an unused node in ExpandIntRes_ADDSUB. NFC

The Hi result is sometimes calculated a different way and this
node goes unused. Defer creation until we know for sure it is neeeded.

The test changes is because the node creation order changed the names
in the debug output.

show more ...

[AArch64][SelectionDAG] Vector splitting and promotion for histogram intrinsic (#103037)

Adds support for wider-than-legal vector types for the histogram
intrinsic (llvm.experimental.vector.histogr

[AArch64][SelectionDAG] Vector splitting and promotion for histogram intrinsic (#103037)

Adds support for wider-than-legal vector types for the histogram
intrinsic (llvm.experimental.vector.histogram.add) by splitting the
vector. Also adds integer promotion for the Inc operand.

show more ...

[ExtendLifetimes] Implement llvm.fake.use to extend variable lifetimes (#86149)

This patch is part of a set of patches that add an `-fextend-lifetimes`
flag to clang, which extends the lifetimes of

[ExtendLifetimes] Implement llvm.fake.use to extend variable lifetimes (#86149)

This patch is part of a set of patches that add an `-fextend-lifetimes`
flag to clang, which extends the lifetimes of local variables and
parameters for improved debuggability. In addition to that flag, the
patch series adds a pragma to selectively disable `-fextend-lifetimes`,
and an `-fextend-this-ptr` flag which functions as `-fextend-lifetimes`
for this pointers only. All changes and tests in these patches were
written by Wolfgang Pieb (@wolfy1961), while Stephen Tozer (@SLTozer)
has handled review and merging. The extend lifetimes flag is intended to
eventually be set on by `-Og`, as discussed in the RFC
here:

https://discourse.llvm.org/t/rfc-redefine-og-o1-and-add-a-new-level-of-og/72850

This patch implements a new intrinsic instruction in LLVM,
`llvm.fake.use` in IR and `FAKE_USE` in MIR, that takes a single operand
and has no effect other than "using" its operand, to ensure that its
operand remains live until after the fake use. This patch does not emit
fake uses anywhere; the next patch in this sequence causes them to be
emitted from the clang frontend, such that for each variable (or this) a
fake.use operand is inserted at the end of that variable's scope, using
that variable's value. This patch covers everything post-frontend, which
is largely just the basic plumbing for a new intrinsic/instruction,
along with a few steps to preserve the fake uses through optimizations
(such as moving them ahead of a tail call or translating them through
SROA).

Co-authored-by: Stephen Tozer <stephen.tozer@sony.com>

show more ...

[SelectionDAG] Use getSignedConstant/getAllOnesConstant.

[SelectionDAG][X86] Use getAllOnesConstant. NFC (#104640)

Part of an effort to make getConstant stricter about implicit truncation
when converting uint64_t to APInt.

[LegalizeTypes] Reuse Op1 and Op2 variables to hold promoted values in PromoteIntRes_ADDSUBSHLSAT. NFC (#102840)

We don't need the original values after we promote them.

[LegalizeTypes][RISCV] Use signed promotion for UADDSAT if that's what the target prefers. (#102842)

As noted in #102781 we can promote UADDSAT if we use sign extend instead
of zero extend.

The

[LegalizeTypes][RISCV] Use signed promotion for UADDSAT if that's what the target prefers. (#102842)

As noted in #102781 we can promote UADDSAT if we use sign extend instead
of zero extend.

The custom handler for RISC-V was using SIGN_EXTEND when the Zbb
extension was enabled. With this change we no longer need the custom
code.

show more ...

[LegalizeTypes][RISCV] Use SExtOrZExtPromotedOperands to promote operands for USUBSAT. (#102781)

It doesn't matter which extend we use to promote the operands. Use
whatever is the most efficient.

[LegalizeTypes][RISCV] Use SExtOrZExtPromotedOperands to promote operands for USUBSAT. (#102781)

It doesn't matter which extend we use to promote the operands. Use
whatever is the most efficient.

The custom handler for RISC-V was using SIGN_EXTEND when the Zbb
extension is enabled so we no longer need that.

show more ...

[LegalizeTypes] Use APInt::getLowBitsSet instead of getAllOnes+zext. NFC

[llvm] Construct SmallVector<SDValue> with ArrayRef (NFC) (#102578)

[DAG] Add legalization handling for ABDS/ABDU (#92576) (REAPPLIED)

Always match ABD patterns pre-legalization, and use TargetLowering::expandABD to expand again during legalization.

abdu(lhs, rhs)

[DAG] Add legalization handling for ABDS/ABDU (#92576) (REAPPLIED)

Always match ABD patterns pre-legalization, and use TargetLowering::expandABD to expand again during legalization.

abdu(lhs, rhs) -> sub(xor(sub(lhs, rhs), usub_overflow(lhs, rhs)), usub_overflow(lhs, rhs))
Alive2: https://alive2.llvm.org/ce/z/dVdMyv

REAPPLIED: Fix regression issue with "abs(ext(x) - ext(y)) -> zext(abd(x, y))" fold failing after type legalization

show more ...

Revert b1234ddbe2652aa7948242a57107ca7ab12fd2f8. "[DAG] Add legalization handling for ABDS/ABDU (#92576)"

Reverting #92576 while we identify a reported regression

[DAG] Add legalization handling for ABDS/ABDU (#92576)

Always match ABD patterns pre-legalization, and use TargetLowering::expandABD to expand again during legalization.

abdu(lhs, rhs) -> sub(xor

[DAG] Add legalization handling for ABDS/ABDU (#92576)

Always match ABD patterns pre-legalization, and use TargetLowering::expandABD to expand again during legalization.

abdu(lhs, rhs) -> sub(xor(sub(lhs, rhs), usub_overflow(lhs, rhs)), usub_overflow(lhs, rhs))
Alive2: https://alive2.llvm.org/ce/z/dVdMyv

show more ...