[CostModel][X86] Improve AVX1/AVX2 v16i32->v16i16/v16i8 truncation costs (PR51972)

Based off worst case btver2 (AVX1) and haswell (AVX2) llvm-mca reports

[CostModel][X86] Increase i64 mul cost from 1 to 2

Only the most recent cpus support really 1cy 64-bit multiplies, and the X64 cost table represents a realistic worst case. The 1cy value was also di

[CostModel][X86] Increase i64 mul cost from 1 to 2

Only the most recent cpus support really 1cy 64-bit multiplies, and the X64 cost table represents a realistic worst case. The 1cy value was also discouraging vectorization when most vXi64 PMULDQ expansions aren't actually slower than scalarization.

Noticed while investigating PR51436.

show more ...

[CostModel][X86] Adjust truncate SSE/AVX legalized costs based on llvm-mca reports.

Update truncation costs based on the worst case costs from the script in D103695.

Move to using legalized types w

[CostModel][X86] Adjust truncate SSE/AVX legalized costs based on llvm-mca reports.

Update truncation costs based on the worst case costs from the script in D103695.

Move to using legalized types wherever possible, which allows us to prune the cost tables.

show more ...

[CostModel][X86] Adjust sext/zext SSE/AVX legalized costs based on llvm-mca reports.

Update costs based on the worst case costs from the script in D103695.

Move to using legalized types wherever po

[CostModel][X86] Adjust sext/zext SSE/AVX legalized costs based on llvm-mca reports.

Update costs based on the worst case costs from the script in D103695.

Move to using legalized types wherever possible, which allows us to prune the cost tables.

show more ...

[CostModel][X86] Improve AVX1/AVX2 truncation costs

Based off the worse case numbers generated by D103695, we were overestimating the cost of a number of vector truncations:

AVX2: v2i32->v2i8, v2i6

[CostModel][X86] Improve AVX1/AVX2 truncation costs

Based off the worse case numbers generated by D103695, we were overestimating the cost of a number of vector truncations:

AVX2: v2i32->v2i8, v2i64->v2i16 + v4i64->v4i32
AVX1: v2i32->v2i8, v4i64->v4i16 + v16i16->v16i8

Once we have a working set of conversion costs, the intention is to cleanup the tables and use legalized types a lot more to reduce the number of entries we currently have.

show more ...

[CostModel][X86] Improve accuracy of sext/zext to 256-bit vector costs on AVX1 targets

Determined from llvm-mca analysis (btver2 vs bdver2 vs sandybridge), the split+extends+concat sequence on AVX1

[CostModel][X86] Improve accuracy of sext/zext to 256-bit vector costs on AVX1 targets

Determined from llvm-mca analysis (btver2 vs bdver2 vs sandybridge), the split+extends+concat sequence on AVX1 capable targets are cheaper than the #ops that the cost was previously based on.

show more ...

[CostModel][X86] Improve accuracy of vXi64 MUL costs on AVX2/AVX512 targets

By llvm-mca analysis, Haswell/Broadwell has the worst v4i64 recip-throughput cost of the AVX2 targets at 6 (vs the current

[CostModel][X86] Improve accuracy of vXi64 MUL costs on AVX2/AVX512 targets

By llvm-mca analysis, Haswell/Broadwell has the worst v4i64 recip-throughput cost of the AVX2 targets at 6 (vs the currently used cost of 8). Similarly SkylakeServer (our only AVX512 target model) implements PMULLQ with an average cost of 1.5 (rounded up to 2.0), and the PMULUDQ-sequence (without AVX512DQ) as a cost of 6.

show more ...

[CostModel][X86] Align v2i64 MUL costs on SSE42+ targets with worst case

Based on worst case of sandybridge (which seems to match nehalem for this SSE sequence) (vs btver2 + bdver2) llvm-mca analysis

[CostModel][X86] Align v4i64 MUL costs on AVX1 targets with worst case

Based on worst case of sandybridge (vs btver2 + bdver2) llvm-mca analysis - which is a lot less than what we were predicting (I

[CostModel][X86] Align v4i64 MUL costs on AVX1 targets with worst case

Based on worst case of sandybridge (vs btver2 + bdver2) llvm-mca analysis - which is a lot less than what we were predicting (I think based off total uop count).

show more ...

[CostModel][X86] Improve v8i32 MUL costs on AVX1 targets to account for slower btver2

BTVER2 has a 2 cycle throughput for v4i32 multiplies (same as SSE41 targets), which is only partially hidden by

[CostModel][X86] Improve v8i32 MUL costs on AVX1 targets to account for slower btver2

BTVER2 has a 2 cycle throughput for v4i32 multiplies (same as SSE41 targets), which is only partially hidden by the subvector extracts/insert when splitting v8i32.

show more ...

[CostModel][X86] Remove unused CHECK prefixes

Allows us to remove the "CHECK: {{^}}" hack and help simplify D91275

[CostModel] fix cost calc bug for sadd/ssub with overflow

As noted in D90554, there's an opcode typo in using an easily
misused cost model API: getCmpSelInstrCost(). Beyond that, the
assumed sequenc

[CostModel] fix cost calc bug for sadd/ssub with overflow

As noted in D90554, there's an opcode typo in using an easily
misused cost model API: getCmpSelInstrCost(). Beyond that, the
assumed sequence of ops is questionable, but that would be
another patch.

My guess is that the x86 test diffs show that we are probably
wrong both before and after this change, so there will be no
practical difference.
As an example, I tried this test which shows a cost of '7'
either way:

define <4 x i32> @sadd(<4 x i32> %va, <4 x i32> %vb) {
%V4I32 = call {<4 x i32>, <4 x i1>} @llvm.sadd.with.overflow.v4i32(<4 x i32> %va, <4 x i32> %vb)
%ov = extractvalue {<4 x i32>, <4 x i1>} %V4I32, 1
%r = extractvalue {<4 x i32>, <4 x i1>} %V4I32, 0
%z = select <4 x i1> %ov, <4 x i32> <i32 42, i32 42, i32 42, i32 42>, <4 x i32> %r
ret <4 x i32> %z
}

$ llc -o - sadd.ll -mattr=avx
vpaddd %xmm1, %xmm0, %xmm2
vpcmpgtd %xmm2, %xmm0, %xmm0
vpxor %xmm0, %xmm1, %xmm0
vblendvps %xmm0, LCPI0_0(%rip), %xmm2, %xmm0a

Differential Revision: https://reviews.llvm.org/D90681

show more ...

[test] Fix some unused check prefixes in test/Analysis/CostModel/X86

[x86] add cost overrides for mul with overflow

I'm assuming the standard size integer instructions for this end up as something like:
mulq %rsi
seto %al

And the 'mul' generally has reciprocal throu

[x86] add cost overrides for mul with overflow

I'm assuming the standard size integer instructions for this end up as something like:
mulq %rsi
seto %al

And the 'mul' generally has reciprocal throughput of 1 on typical implementations
(higher latency, but that's not handled here).
The default costs may end up much higher than that, and that's what we see in the test diffs.

Vector types are left as a 'TODO'.

Differential Revision: https://reviews.llvm.org/D90431

show more ...

[CostModel][X86][ARM] Teach default implementation of getCastInstrCost to not add a split/join cost if source type and the destination type both have a SplitVector action

If both the source and the

[CostModel][X86][ARM] Teach default implementation of getCastInstrCost to not add a split/join cost if source type and the destination type both have a SplitVector action

If both the source and the destination need to be split then the two halves of the split operation are completely independent and don't need to be split or joined. So we don't need to assess a cost for the split or join.

Differential Revision: https://reviews.llvm.org/D79111

show more ...

[X86] Lower the cost of v4i64->v4i32 truncate with avx512.

We use the vpmovqd instruction which is a single uop. So
the cost should be 1.

[X86] Lower the cost of v4i64->v4i32 and v8i64->v8i32 truncate with AVX

We generate much better code these days than we used to. And we use the same sequence for AVX1 and AVX2 for these

For v4i64->

[X86] Lower the cost of v4i64->v4i32 and v8i64->v8i32 truncate with AVX

We generate much better code these days than we used to. And we use the same sequence for AVX1 and AVX2 for these

For v4i64->v4i32 we generate:
vextractf128 xmm1, ymm0, 1
vshufps xmm0, xmm0, xmm1, 136 # xmm0 = xmm0[0,2],xmm1[0,2]

And for v8i64->v8i32 we generate:
vperm2f128 ymm2, ymm0, ymm1, 49 # ymm2 = ymm0[2,3],ymm1[2,3]
vinsertf128 ymm0, ymm0, xmm1, 1
vshufps ymm0, ymm0, ymm2, 136 # ymm0 = ymm0[0,2],ymm2[0,2],ymm0[4,6],ymm2[4,6]

Differential Revision: https://reviews.llvm.org/D79109

show more ...

[X86][CostModel] Update costs for vector truncate with avx512f/avx512bw.

All avx512 truncate instructions except vXi64->vXi32 are 2 uops
on port 5. So raise their costs to 2. Except when we have an

[X86][CostModel] Update costs for vector truncate with avx512f/avx512bw.

All avx512 truncate instructions except vXi64->vXi32 are 2 uops
on port 5. So raise their costs to 2. Except when we have an
earlier faster sequence like pshufb for 128 bit input vectors.

Add a lower cost of 3 v16i16->v16i8 with avx512f where we can
extend to v16i32 then truncate. And a cost of 2 for avx512bw with
and without avx512vl. There we can use vpmovwb with either a ymm
or zmm input. Both of these beat masking, splitting, and using
packuswb which is our avx/avx2 codegen.

show more ...

[X86] Make v32i16/v64i8 legal types without avx512bw. Use custom splitting instead.

This moves v32i16/v64i8 to a model consistent with how we
treat integer types with avx1.

This does change the ABI

[X86] Make v32i16/v64i8 legal types without avx512bw. Use custom splitting instead.

This moves v32i16/v64i8 to a model consistent with how we
treat integer types with avx1.

This does change the ABI for types vXi16/vXi8 vectors larger than
512 bits to pass in multiple zmms instead of multiple ymms. We'd
already hacked some code to make v64i8/v32i16 pass in zmm.

Cost model is still a bit of a mess. In some place I tried to
match existing behavior. But really we need to account for
splitting and concating costs. Cost model for shuffles is
especially pessimistic.

Differential Revision: https://reviews.llvm.org/D76212

show more ...

[X86] Add a non-zero cost for truncating v32i16->v32i8 on avx512bw.

[CostModel][X86] Try to check against common prefixes before using target-specific cpu checks

SLM/GLM is still a mess so not all of them have been updated yet.

[X86] Fix the cost model for v16i16->v16i32 zero_extend/sign_extend with AVX2

We seem to be inheriting the cost from sse4.1. But if we have 256-bit registers we should be able to do this with just o

[X86] Fix the cost model for v16i16->v16i32 zero_extend/sign_extend with AVX2

We seem to be inheriting the cost from sse4.1. But if we have 256-bit registers we should be able to do this with just one extract to split the 16i16 and two v8i16->v8i32 operations so our cost should be 3 not 4.

Differential Revision: https://reviews.llvm.org/D73646

show more ...

[CostModel][X86] Fix SLM <2 x i64> icmp costs

SLM is 2 x slower for <2 x i64> comparison ops than other vector types, we should account for this like we do for SLM <2 x i64> add/sub/mul costs.

This

[CostModel][X86] Fix SLM <2 x i64> icmp costs

SLM is 2 x slower for <2 x i64> comparison ops than other vector types, we should account for this like we do for SLM <2 x i64> add/sub/mul costs.

This should remove some of the SLM codegen diffs in D43582

llvm-svn: 372954

show more ...

[Cost][X86] Add v2i64 truncation costs

We are missing costs for a lot of truncation cases, I'm hoping to address all the 'zero cost' cases in trunc.ll

I thought this was a vector widening side effe

[Cost][X86] Add v2i64 truncation costs

We are missing costs for a lot of truncation cases, I'm hoping to address all the 'zero cost' cases in trunc.ll

I thought this was a vector widening side effect, but even before this we had some interesting LV decisions (notably over indvars) being made due to these zero costs.

llvm-svn: 372498

show more ...

[TTI] Add generic cost model for smul/umul overflow intrinsics

Based off smul/umul fixed costs and the implementation in TargetLowering::expandMULO.

llvm-svn: 354784