#
ec402a2e |
| 27-Jan-2024 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Implement cloning of VPlans. (#73158)
This patch implements cloning for VPlans and recipes. Cloning is used in
the epilogue vectorization path, to clone the VPlan for the main vector
loop.
[VPlan] Implement cloning of VPlans. (#73158)
This patch implements cloning for VPlans and recipes. Cloning is used in
the epilogue vectorization path, to clone the VPlan for the main vector
loop. This means we won't re-use a VPlan when executing the VPlan for
the epilogue vector loop, which in turn will enable us to perform
optimizations based on UF & VF.
show more ...
|
#
731c2049 |
| 26-Jan-2024 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Relax IV user assertion after 0ab539f for epilogue vec.
After 0ab539fd6748adf2f638e10514dd9419597d8863, the canonical IV in the epilogue vector loop may be used by a trunc. Relax the corresp
[VPlan] Relax IV user assertion after 0ab539f for epilogue vec.
After 0ab539fd6748adf2f638e10514dd9419597d8863, the canonical IV in the epilogue vector loop may be used by a trunc. Relax the corresponding assert.
This should fix some build-bot failures, including https://lab.llvm.org/buildbot/#/builders/187/builds/14113 https://lab.llvm.org/buildbot/#/builders/98/builds/32350 https://lab.llvm.org/buildbot/#/builders/239/builds/5473
show more ...
|
Revision tags: llvmorg-19-init |
|
#
3683852d |
| 21-Jan-2024 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Use replaceUsesWithIf in replaceAllUseswith and add comment (NFCI).
Follow-up to post-commit commens for b1bfe221e6.
|
#
241fe837 |
| 04-Jan-2024 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Introduce ComputeReductionResult VPInstruction opcode. (#70253)
This patch introduces a new ComputeReductionResult opcode to compute the
final reduction result in the middle block. The code
[VPlan] Introduce ComputeReductionResult VPInstruction opcode. (#70253)
This patch introduces a new ComputeReductionResult opcode to compute the
final reduction result in the middle block. The code from fixReduction
has been moved to ComputeReductionResult, after some earlier cleanup
changes to model parts of fixReduction explicitly elsewhere as needed.
The recipe may be broken down further in the future.
Note that the phi nodes to merge the reduction result from the trip
count check and the middle block, to be used as resume value for the
scalar remainder loop are also generated based on
ComputeReductionResult.
Once we have a VPValue for the reduction result, this can also be
modeled explicitly and moved out of the recipe.
show more ...
|
#
b1bfe221 |
| 15-Dec-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Remove unneeded getNumUsers calls in replaceAllUsesWith (NFC).
As suggested post-commit for a00227197, replace unnecessary getNumUsers calls by boolean variable to indicate if users changed.
[VPlan] Remove unneeded getNumUsers calls in replaceAllUsesWith (NFC).
As suggested post-commit for a00227197, replace unnecessary getNumUsers calls by boolean variable to indicate if users changed. Note that this also requires an early exit to detect the case where a value is replaced by itself.
show more ...
|
#
a5891fa4 |
| 08-Dec-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Initial modeling of VF * UF as VPValue. (#74761)
This patch starts initial modeling of VF * UF in VPlan.
Initially, introduce a dedicated VFxUF VPValue, which is then
populated during VPla
[VPlan] Initial modeling of VF * UF as VPValue. (#74761)
This patch starts initial modeling of VF * UF in VPlan.
Initially, introduce a dedicated VFxUF VPValue, which is then
populated during VPlan::prepareToExecute. Initially, the VF * UF
applies only to the main vector loop region. Once we extend the
scope of VPlan in the future, we may want to associate different VFxUFs
with different vector loop regions (e.g. the epilogue vector loop)
This allows explicitly parameterizing recipes that rely on the
VF * UF, like the canonical induction increment. At the moment, this
mainly helps to avoid generating some duplicated calls to vscale with
scalable vectors. It should also allow using EVL as induction increments
explicitly in D99750. Referring to VF * UF is also needed in other
places that we plan to migrate to VPlan, like the minimum trip count
check during skeleton creation.
The first version creates the value for VF * UF directly in
prepareToExecute to limit the scope of the patch. A follow-on patch will
model VF * UF computation explicitly in VPlan using recipes.
Moved from Phabricator (https://reviews.llvm.org/D157322)
show more ...
|
#
99aa5311 |
| 04-Dec-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Add missing output of live-ins to VPlan dot printing.
Split off live-in printing to VPlan::printLiveIns and use it to print Live-ins when printing in the DOT format.
|
Revision tags: llvmorg-17.0.6 |
|
#
906f5982 |
| 23-Nov-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Remove dead IsEpilogueVec argument from prepareToExecute (NFC).
|
Revision tags: llvmorg-17.0.5 |
|
#
34c2dcd5 |
| 12-Nov-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Move initial skeleton construction to createInitialVPlan. (NFC)
This patch moves creating the middle VPBBs and an initial empty vector loop region for the top-level loop to createInitialVPl
[VPlan] Move initial skeleton construction to createInitialVPlan. (NFC)
This patch moves creating the middle VPBBs and an initial empty vector loop region for the top-level loop to createInitialVPlan.
This consolidates code to create the initial VPlan skeleton and enables adding other bits outside the main region during initial VPlan construction. In particular, D150398 will add the exit check & branch to the middle block.
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D158333
show more ...
|
#
a0022719 |
| 06-Nov-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Add VPValue::replaceUsesWithIf (NFCI).
Add replaceUsesWithIf helper and use it in a few places.
|
Revision tags: llvmorg-17.0.4 |
|
#
3af0ff99 |
| 22-Oct-2023 |
Kazu Hirata <kazu@google.com> |
[llvm] Stop including llvm/ADT/DepthFirstIterator.h (NFC)
Identified with misc-include-cleaner.
|
Revision tags: llvmorg-17.0.3, llvmorg-17.0.2 |
|
#
97687b7a |
| 25-Sep-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Add active-lane-mask as VPlan-to-VPlan transformation.
This patch updates the mask creation code to always create compares of the form (ICMP_ULE, wide canonical IV, backedge-taken-count) up
[VPlan] Add active-lane-mask as VPlan-to-VPlan transformation.
This patch updates the mask creation code to always create compares of the form (ICMP_ULE, wide canonical IV, backedge-taken-count) up front when tail folding and introduce active-lane-mask as later transformation.
This effectively makes (ICMP_ULE, wide canonical IV, backedge-taken-count) the canonical form for tail-folding early on. Introducing more specific active-lane-mask recipes is treated as a VPlan-to-VPlan optimization.
This has the advantage of keeping the logic (and complexity) of introducing active-lane-mask recipes in a single place, instead of spreading the logic out across multiple functions. It also simplifies initial VPlan construction and enables treating introducing EVL as similar optimization.
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D158779
show more ...
|
Revision tags: llvmorg-17.0.1, llvmorg-17.0.0 |
|
#
168e23c7 |
| 05-Sep-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Remove reference to Instr when setting debug loc. (NFCI)
This allows untangling references to underlying IR for various recipes.
|
Revision tags: llvmorg-17.0.0-rc4 |
|
#
19d286bc |
| 03-Sep-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Assert that inst isnt' a debug or pseudo inst (NFCI).
Debug and pseudo instructions aren't modeled in VPlan. Turn a check into an assertion. This will help removing the direct use of Inst he
[VPlan] Assert that inst isnt' a debug or pseudo inst (NFCI).
Debug and pseudo instructions aren't modeled in VPlan. Turn a check into an assertion. This will help removing the direct use of Inst here in the future.
show more ...
|
Revision tags: llvmorg-17.0.0-rc3 |
|
#
e18a547c |
| 08-Aug-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Fold if into return in prepareToExecute assertion (NFC).
Independent simplification suggested in D157194.
|
#
af635a55 |
| 08-Aug-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Model wrap flags directly, remove *NUW opcodes (NFC)
Model wrap flags directly using VPRecipeWithIRFlags and clean up the duplicated *NUW opcodes.
D157144 will build on this and also model
[VPlan] Model wrap flags directly, remove *NUW opcodes (NFC)
Model wrap flags directly using VPRecipeWithIRFlags and clean up the duplicated *NUW opcodes.
D157144 will build on this and also model FMFs for VPInstruction.
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D157194
show more ...
|
Revision tags: llvmorg-17.0.0-rc2 |
|
#
deec9e76 |
| 03-Aug-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Move VPTransformState::get() to VPlan.cpp (NFC).
The last dependency of code defined in LoopVectorize.cpp has been removed a while ago. Move VPTransformState::get() to VPlan.cpp where other
[VPlan] Move VPTransformState::get() to VPlan.cpp (NFC).
The last dependency of code defined in LoopVectorize.cpp has been removed a while ago. Move VPTransformState::get() to VPlan.cpp where other members are also defined.
show more ...
|
Revision tags: llvmorg-17.0.0-rc1, llvmorg-18-init |
|
#
d1d0e135 |
| 30-Jun-2023 |
Florian Hahn <flo@fhahn.com> |
[LV] Move packScalarIntoVectorValue to VPTransformState (NFC).
This moves packScalarIntoVectorValue from ILV to the more approriate VPTransformState.
|
#
b0abd489 |
| 17-Jun-2023 |
Elliot Goodrich <elliotgoodrich@gmail.com> |
[llvm] Add missing StringExtras.h includes
In preparation for removing the `#include "llvm/ADT/StringExtras.h"` from the header to source file of `llvm/Support/Error.h`, first add in all the missing
[llvm] Add missing StringExtras.h includes
In preparation for removing the `#include "llvm/ADT/StringExtras.h"` from the header to source file of `llvm/Support/Error.h`, first add in all the missing includes that were previously included transitively through this header.
show more ...
|
Revision tags: llvmorg-16.0.6, llvmorg-16.0.5 |
|
#
96686796 |
| 22-May-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Move live-out printing to VPLiveOut::print (NFC).
Preparation for D150398. This brings live-out printing in line with how printing for recipes is handled.
|
Revision tags: llvmorg-16.0.4, llvmorg-16.0.3, llvmorg-16.0.2 |
|
#
9272d0f0 |
| 17-Apr-2023 |
Hongtao Yu <hoy@fb.com> |
[PseudoProbe] Clean up dwarf discriminator and avoid duplicating factor.
A pseudo probe is created with dwarf line information shared with its nearest instruction. If the instruction comes with a dw
[PseudoProbe] Clean up dwarf discriminator and avoid duplicating factor.
A pseudo probe is created with dwarf line information shared with its nearest instruction. If the instruction comes with a dwarf discriminator, it will be shared with the probe as well. This can confuse the later FS-AFDO discriminator assignment pass. To fix this, I'm cleaning up the discriminator fields for probes when they are inserted.
I also notice another possibility to change the discriminator field of pseudo probes in the pipeline before the FS discriminator assignment pass. That is the loop unroller, which assigns duplication factor to instruction being vectorized. I'm disabling that for pseudo probe intrinsics specifically, also for callsites with probes.
Reviewed By: wenlei
Differential Revision: https://reviews.llvm.org/D148569
show more ...
|
#
e3afe0b8 |
| 05-May-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Add VPWidenCastRecipe, split off from VPWidenRecipe (NFCI).
To generate cast instructions, the result type is needed. To allow creating widened casts without underlying instruction, introduc
[VPlan] Add VPWidenCastRecipe, split off from VPWidenRecipe (NFCI).
To generate cast instructions, the result type is needed. To allow creating widened casts without underlying instruction, introduce a new VPWidenCastRecipe that also holds the result type.
This functionality will be used in a follow-up patch to implement truncateToMinimalBitwidths as VPlan-to-VPlan transform.
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D149081
show more ...
|
#
147a5614 |
| 04-May-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Clean up preheader block after b85a402dd899fc.
Fix a leak introduced in b85a402dd899fc and flagged by LSan https://lab.llvm.org/buildbot#builders/5/builds/33452
|
#
b85a402d |
| 04-May-2023 |
Florian Hahn <flo@fhahn.com> |
[VPlan] Introduce new entry block to VPlan for early SCEV expansion.
This patch adds a new preheader block the VPlan to place SCEV expansions expansions like the trip count. This preheader block is
[VPlan] Introduce new entry block to VPlan for early SCEV expansion.
This patch adds a new preheader block the VPlan to place SCEV expansions expansions like the trip count. This preheader block is disconnected at the moment, as the bypass blocks of the skeleton are not yet modeled in VPlan.
The preheader block is executed before skeleton creation, so the SCEV expansion results can be used during skeleton creation. At the moment, the trip count expression and induction steps are expanded in the new preheader. The remainder of SCEV expansions will be moved gradually in the future.
D147965 will update skeleton creation to use the steps expanded in the pre-header to fix #58811.
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D147964
show more ...
|
#
79692750 |
| 04-May-2023 |
Florian Hahn <flo@fhahn.com> |
[LV] Use VPValue for SCEV expansion in fixupIVUsers.
The step is already expanded in the VPlan. Use this expansion instead. This is a step towards modeling fixing up IV users in VPlan.
It also fix
[LV] Use VPValue for SCEV expansion in fixupIVUsers.
The step is already expanded in the VPlan. Use this expansion instead. This is a step towards modeling fixing up IV users in VPlan.
It also fixes a crash casued by SCEV-expanding the Step expression in fixupIVUsers, where the IR is in an incomplete state
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D147963
show more ...
|