[VectorCombine] foldPermuteOfBinops - match identity shuffles only if they match the destination type

Fixes regression identified after #122118

[VectorCombine] foldPermuteOfBinops - ensure potential identity mask isn't length changing.

VectorCombine: teach foldExtractedCmps about samesign (#122883)

Follow up on 4a0d53a (PatternMatch: migrate to CmpPredicate) to get rid
of one of the FIXMEs it introduced by replacing a predicate c

VectorCombine: teach foldExtractedCmps about samesign (#122883)

Follow up on 4a0d53a (PatternMatch: migrate to CmpPredicate) to get rid
of one of the FIXMEs it introduced by replacing a predicate comparison
with CmpPredicate::getMatching.

show more ...

[VectorCombine] foldPermuteOfBinops - fold "shuffle (binop (shuffle, other)), undef" --> "binop (shuffle), (shuffle)". (#122118)

foldPermuteOfBinops currently requires both binop operands to be oneu

[VectorCombine] foldPermuteOfBinops - fold "shuffle (binop (shuffle, other)), undef" --> "binop (shuffle), (shuffle)". (#122118)

foldPermuteOfBinops currently requires both binop operands to be oneuse shuffles to fold the shuffles across the binop, but there will be cases where its still profitable to fold across the binop with only one foldable shuffle.

show more ...

[VectorCombine] Use getInstructionCost to cost Shuffle. (#122068)

This allows it to produce a more accurate cost for the shuffle, using
the more accurate calls to getShuffleCost in getInstructionCo

[VectorCombine] Use getInstructionCost to cost Shuffle. (#122068)

This allows it to produce a more accurate cost for the shuffle, using
the more accurate calls to getShuffleCost in getInstructionCost. It
helps fix some of the regressions from vector combine a little while
ago, now that we have better subvector extract costs.

show more ...

[CostModel][X86] getVectorInstrCost - correctly cost v4f32 insertelement into index 0

This is just the MOVSS instruction (SSE41 INSERTPS is still necessary for index != 0)

This exposed an issue in

[CostModel][X86] getVectorInstrCost - correctly cost v4f32 insertelement into index 0

This is just the MOVSS instruction (SSE41 INSERTPS is still necessary for index != 0)

This exposed an issue in VectorCombine::foldInsExtFNeg - we need to use the more general SK_PermuteTwoSrc shuffle kind to allow getShuffleCost to match other shuffle kinds (not just SK_Select).

show more ...

[VectorCombine] Remove superfluous whitespace from debug log comment. NFC.

[VectorCombine] foldInsExtVectorToShuffle - ignore shuffle costs for 'identity' insertion masks

<u,1,u,u> 'inplace' single src shuffles can be treated as free identity shuffles - ignore any shuffle

[VectorCombine] foldInsExtVectorToShuffle - ignore shuffle costs for 'identity' insertion masks

<u,1,u,u> 'inplace' single src shuffles can be treated as free identity shuffles - ignore any shuffle cost (similar to what we already do in other folds like foldShuffleOfShuffles) - eventually getShuffleCost should just return TCC_Free in these cases but in a lot of the targets' shuffle cost logic this currently ends up treated as a generic SK_PermuteSingleSrc.

We still want to generate the shuffle as it will help further shuffle folds with the additional PoisonMaskElem 'undemanded' elements.

show more ...

[VectorCombine] foldShuffleOfBinops - fold shuffle(binop(shuffle(x),shuffle(z)),binop(shuffle(y),shuffle(w)) -> binop(shuffle(x,z),shuffle(y,w)) (#120984)

Some patterns (in particular horizontal sty

[VectorCombine] foldShuffleOfBinops - fold shuffle(binop(shuffle(x),shuffle(z)),binop(shuffle(y),shuffle(w)) -> binop(shuffle(x,z),shuffle(y,w)) (#120984)

Some patterns (in particular horizontal style patterns) can end up with shuffles straddling both sides of a binop/cmp.

Where individually the folds aren't worth it, by merging the (oneuse) shuffles we can notably reduce the net instruction count and cost.

One of the final steps towards finally addressing #34072

show more ...

[VectorCombine] eraseInstruction - ensure we reattempt to fold other users of an erased instruction's operands (REAPPLIED)

As we're reducing the use count of the operands its more likely that they w

[VectorCombine] eraseInstruction - ensure we reattempt to fold other users of an erased instruction's operands (REAPPLIED)

As we're reducing the use count of the operands its more likely that they will now fold, as they were previously being prevented by a m_OneUse check, or the cost of retaining the extra instruction had been too high.

This is necessary for some upcoming patches, although the only change so far is instruction ordering as it allows some SSE folds of 256/512-bit with 128-bit subvectors to occur earlier in foldShuffleToIdentity as the subvector concats are free.

Reapplied with a fix for foldSingleElementStore/scalarizeLoadExtract which were replacing/removing memory operations - we need to ensure that the worklist is populated in the correct order so all users of the old memory operations are erased first, so there are no remaining users of the loads when its time to remove them as well.

Pulled out of #120984

show more ...

[VectorCombine] replaceValue - add "VC: Replacing" debug message to help the log show replacement for old/new.

[VectorCombine] scalarizeLoadExtract - consistently use LoadInst and ExtractElementInst specific operand getters. NFC

Noticed while investigating the hung builds reported after af83093933ca73bc82c33

[VectorCombine] scalarizeLoadExtract - consistently use LoadInst and ExtractElementInst specific operand getters. NFC

Noticed while investigating the hung builds reported after af83093933ca73bc82c33130f8bda9f1ae54aae2

show more ...

Revert af83093933ca73bc82c33130f8bda9f1ae54aae2 "[VectorCombine] eraseInstruction - ensure we reattempt to fold other users of an erased instruction's operands"

Reports of hung builds, but I don't h

Revert af83093933ca73bc82c33130f8bda9f1ae54aae2 "[VectorCombine] eraseInstruction - ensure we reattempt to fold other users of an erased instruction's operands"

Reports of hung builds, but I don't have time to investigate at the moment.

show more ...

[VectorCombine] eraseInstruction - ensure we reattempt to fold other users of an erased instruction's operands

As we're reducing the use count of the operands its more likely that they will now fold

[VectorCombine] eraseInstruction - ensure we reattempt to fold other users of an erased instruction's operands

As we're reducing the use count of the operands its more likely that they will now fold, as they were previously being prevented by a m_OneUse check, or the cost of retaining the extra instruction had been too high.

This is necessary for some upcoming patches, although the only change so far is instruction ordering as it allows some SSE folds of 256/512-bit with 128-bit subvectors to occur earlier in foldShuffleToIdentity as the subvector concats are free.

Pulled out of #120984

show more ...

[VectorCombine] foldShuffleOfBinops - only accept exact matching cmp predicates

m_SpecificCmp allowed equivalent predicate+flags which don't necessarily work after being folded from "shuffle (cmpop)

[VectorCombine] foldShuffleOfBinops - only accept exact matching cmp predicates

m_SpecificCmp allowed equivalent predicate+flags which don't necessarily work after being folded from "shuffle (cmpop), (cmpop)" into "cmpop (shuffle), (shuffle)"

Fixes #121110

show more ...

[VectorCombine] foldInsExtVectorToShuffle - inserting into a poison base vector can be modelled as a single src shuffle

We already canonicalized an undef base vector to the RHS to improve further fo

[VectorCombine] foldInsExtVectorToShuffle - inserting into a poison base vector can be modelled as a single src shuffle

We already canonicalized an undef base vector to the RHS to improve further folding, this extends this to improve the shuffle cost estimate of the single src shuffle

show more ...

[VectorCombine] foldShuffleOfShuffles - fold "shuffle (shuffle x, y, m1), (shuffle y, x, m2)" -> "shuffle x, y, m3" (#120959)

foldShuffleOfShuffles currently only folds unary shuffles to ensure we d

[VectorCombine] foldShuffleOfShuffles - fold "shuffle (shuffle x, y, m1), (shuffle y, x, m2)" -> "shuffle x, y, m3" (#120959)

foldShuffleOfShuffles currently only folds unary shuffles to ensure we don't end up with a merged shuffle with more than 2 sources, but this prevented cases where both shuffles were sharing sources.

This patch generalizes the merge process to find up to 2 sources as it merges with the inner shuffles, it also moves the undef/poison handling stages into the merge loop as well.

Fixes #120764

show more ...

[VectorCombine] foldShuffleToIdentity - add debug message for match

Helps with debugging to show to that the fold found the match.

[VectorCombine] foldConcatOfBoolMasks - add debug message for match + cost-comparison

Helps with debugging to show to that the fold found the match, and shows the old + new costs to indicate whether

[VectorCombine] foldConcatOfBoolMasks - add debug message for match + cost-comparison

Helps with debugging to show to that the fold found the match, and shows the old + new costs to indicate whether the fold was/wasn't profitable.

show more ...

[VectorCombine] Add "VC: Erasing" debug message to help the log show when dead WorkList instructions are erased.

[VectorCombine] foldBitcastShuffle - add debug message for match + cost-comparison

Helps with debugging to show to that the fold found the match, and shows the old + new costs to indicate whether th

[VectorCombine] foldBitcastShuffle - add debug message for match + cost-comparison

Helps with debugging to show to that the fold found the match, and shows the old + new costs to indicate whether the fold was/wasn't profitable.

show more ...

[VectorCombine] Add fp cast handling for shuffletoidentity (#120641)

This fixes some regressions from recent changes to vector combine in
#120216. It allows shuffleToIdentity to look through fp cas

[VectorCombine] Add fp cast handling for shuffletoidentity (#120641)

This fixes some regressions from recent changes to vector combine in
#120216. It allows shuffleToIdentity to look through fp casts as other
casts, and makes sure mismatching vector types in splats and casts do
not block the transform, as only the lanes should matter.

show more ...

[VectorCombine] Add "VC: Visiting" debug message to help the log show the instruction folding order.

[VectorCombine] Add "VECTORCOMBINE on <FUNCTION_NAME>" title debug message to help finding vectorcombine stages in the debug log

[VectorCombine] foldInsExtVectorToShuffle - add debug message for match + cost-comparison

Helps with debugging to show to that the fold found the match, and shows the old + new costs to indicate whe

[VectorCombine] foldInsExtVectorToShuffle - add debug message for match + cost-comparison

Helps with debugging to show to that the fold found the match, and shows the old + new costs to indicate whether the fold was/wasn't profitable.

show more ...