[CostModel][X86] Add missing AVX512 vector mul overflow intrinsic costs

Fix regressions in #100519

[CostModel][X86] Improve vector mul overflow intrinsic costs

[CostModel][X86] Improve add/sub/mul overflow intrinsic costs

Noticed due to x86 changes in #97463

[CostModel][X86] Tweak SSE2 v2i64 multiply costs based off D46276 script

It looks like we were trying to account for SLM costs, which are actually handled separately

Fixes #62969

[X86] Fix Icelake VPMULLQ zmm pipes and adjust AVX512DQ v8i64 mul costs to match worse case

Icelake PMULLQ throughput regressed cf SkylakeServer as its Pipe0 only

Confirmed with Intel SOM, Agner an

[X86] Fix Icelake VPMULLQ zmm pipes and adjust AVX512DQ v8i64 mul costs to match worse case

Icelake PMULLQ throughput regressed cf SkylakeServer as its Pipe0 only

Confirmed with Intel SOM, Agner and instlatx64

show more ...

[CostModel][X86] Add CostKinds handling for vector integer comparisons

These were based off a mixture of vector integer add/sub costs and the numbers from the 'cost-tables vs llvm-mca' script from D

[CostModel][X86] Add CostKinds handling for vector integer comparisons

These were based off a mixture of vector integer add/sub costs and the numbers from the 'cost-tables vs llvm-mca' script from D103695 - the extra costs for different predicates are still proving tricky to implement, but I've gotten most costs to within +/1 now - the AVX512 are tricky as we still don't handle predicate results properly, so most of these were done by hand.

show more ...

[CostModel][X86] Add CostKinds handling for vector shift by uniform/constuniform ops

Vector shift by const uniform is the cheapest shift instruction we have, non-const uniform have a marginally high

[CostModel][X86] Add CostKinds handling for vector shift by uniform/constuniform ops

Vector shift by const uniform is the cheapest shift instruction we have, non-const uniform have a marginally higher cost - some targets 'splat' the amount internally to use the shift-per-element instruction, others see a higher cost for the explicit zeroing of the upper bits for the (64-bit) shift amount.

This was achieved with an updated version of the 'cost-tables vs llvm-mca' script D103695 (I'll update the patch soon for reference)

show more ...

[CostModel][X86] Fix throughput costs for AVX512BW v32i16 shifts

Fixes regression from a931dbfbd30754cf39897037a223eee60ae9e855

[CostModel][X86] Merge AVX512BW vXi8/vXi16 shifts into default AVX512BW cost table

We only need to handle the uniform cases early

[CostModel][X86] Add CostKinds handling for mul ops

This was achieved using the 'cost-tables vs llvm-mca' script D103695

Also fix a missing pmullw v16i16 half-rate throughput as znver1 double-pumps

[CostModel][X86] Add CostKinds handling for mul ops

This was achieved using the 'cost-tables vs llvm-mca' script D103695

Also fix a missing pmullw v16i16 half-rate throughput as znver1 double-pumps - matches numbers from AMD SoG + Agner

show more ...

[CostModel][X86] Adjust mul v4i32/v8i32 throughput cost

Based off the numbers from AMD SoG + Agner - vXi32 are both half-rate, and znver1 double-pumps the v8i32 op

We should have caught this earlie

[CostModel][X86] Adjust mul v4i32/v8i32 throughput cost

Based off the numbers from AMD SoG + Agner - vXi32 are both half-rate, and znver1 double-pumps the v8i32 op

We should have caught this earlier as many Intel models have half-rate pmulld already :-(

show more ...

[CostModel][X86] Account for add/sub 512-bit vector splitting costs on non-AVX512BW targets

[CostModel][X86] Cleanup arithmetic test triples

Just specify the triple inside the RUN command (to make i686 support much easier!), and consistently use x86_64-- generic triple

[CostModel][X86] Check for 'null op' truncations

If the legalized src/dst types are the same, assume the "truncation" is free.

This fixes some edge cases such as mul lo/hi ops and bool vectors whic

[CostModel][X86] Check for 'null op' truncations

If the legalized src/dst types are the same, assume the "truncation" is free.

This fixes some edge cases such as mul lo/hi ops and bool vectors which will get legalized back to legal vector widths

show more ...

[CostModel][X86] Reduce cost of v2i64 icmp base cost on SSE2 targets

Based off the script from D103695, we were exaggerating the cost of the v2i64 comparison expansion using instruction count instea

[CostModel][X86] Reduce cost of v2i64 icmp base cost on SSE2 targets

Based off the script from D103695, we were exaggerating the cost of the v2i64 comparison expansion using instruction count instead of effective throughput

show more ...

[CostModel][X86] Update RUN -passes=* to double quotes to appease update scripts on windows

[test] Migrate '-analyze -cost-model' to '-passes=print<cost-model>'

[CostModel][X86] Add i64 mul cost for avx512 as 1cy

i64 mul cost is 1cy for all cpu that support avx512. Currently
all X86 cpu uses i64 mul cost in X64 cost table which is not
true for cpu that supp

[CostModel][X86] Add i64 mul cost for avx512 as 1cy

i64 mul cost is 1cy for all cpu that support avx512. Currently
all X86 cpu uses i64 mul cost in X64 cost table which is not
true for cpu that support avx512 (skx, icx).

Reviewed By: pengfei, RKSimon

Differential Revision: https://reviews.llvm.org/D115016

show more ...

[X86][Costmodel] `trunc v32i16 to v64i8` can appear after legalization, cost is same as for `trunc v32i16 to v32i8`

Some of the costs get larger here,
but i suppose that makes sense since we'd previ

[X86][Costmodel] `trunc v32i16 to v64i8` can appear after legalization, cost is same as for `trunc v32i16 to v32i8`

Some of the costs get larger here,
but i suppose that makes sense since we'd previously query
scalarization costs that may not be really representative of the reality.

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D113852

show more ...

[X86][Costmodel] `trunc v16i32 to v32i16` can appear after legalization, cost is same as for `trunc v16i32 to v16i16`

This was noticed in D113609, hopefully it unblocks that patch.
There are likely

[X86][Costmodel] `trunc v16i32 to v32i16` can appear after legalization, cost is same as for `trunc v16i32 to v16i16`

This was noticed in D113609, hopefully it unblocks that patch.
There are likely other similar problems.

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D113842

show more ...

[CostModel][TTI] Replace BAD_ICMP_PREDICATE with ICMP_NE for generic smulo/umulo cost expansion

Match the predicate used in TargetLowering::expandMULO to detect overflow

[CostModel][TTI] Fix ops used for generic smulo/umulo cost expansion

Fix copy+pasta that was checking for smul_fix instead of smul_with_overflow to detected signed values.

The LShr is performed on

[CostModel][TTI] Fix ops used for generic smulo/umulo cost expansion

Fix copy+pasta that was checking for smul_fix instead of smul_with_overflow to detected signed values.

The LShr is performed on the extended type as we use it to truncate+extract the upper/hi bits of the extended multiply.

More closely matches the default expansion from TargetLowering::expandMULO

show more ...

[CostModel][TTI] Replace BAD_ICMP_PREDICATE with ICMP_ULT/UGT for generic uadd/usubo cost expansion

Match the predicates used in TargetLowering::expandUADDSUBO

[CostModel][X86] getCmpSelInstrCost - treat BAD_PREDICATEs the same as the worst case cost predicates for ICMP/FCMP instructions

As suggested on D111024, we should treat getCmpSelInstrCost calls wit

[CostModel][X86] getCmpSelInstrCost - treat BAD_PREDICATEs the same as the worst case cost predicates for ICMP/FCMP instructions

As suggested on D111024, we should treat getCmpSelInstrCost calls without a specific predicate as matching the worst case predicate cost.

These regressions will be addressed with a mixture of D111024 and fixing other specific getCmpSelInstrCost calls to have realistic predicates.

show more ...

[CostModel] Update default cost model for sadd/ssub overflow to match TargetLowering

The expansion for these was updated in https://reviews.llvm.org/D47927 but the cost model was not adjusted.

I be

[CostModel] Update default cost model for sadd/ssub overflow to match TargetLowering

The expansion for these was updated in https://reviews.llvm.org/D47927 but the cost model was not adjusted.

I believe the cost model was also incorrect for the old expansion.
The expansion prior to D47927 used 3 icmps using LHS, RHS, and Result
to calculate theirs signs. Then 2 icmps to compare the signs. Followed
by an And. The previous cost model was using 3 icmps and 2 selects.
Digging back through git blame, those 2 selects in the cost model used to
be 2 icmps, but were changed in https://reviews.llvm.org/D90681

Differential Revision: https://reviews.llvm.org/D110739

show more ...