[IR] Replace of PointerType::get(Type) with opaque version (NFC) (#123617)

In accordance with https://github.com/llvm/llvm-project/issues/123569

In order to keep the patch at reasonable size, this

[IR] Replace of PointerType::get(Type) with opaque version (NFC) (#123617)

In accordance with https://github.com/llvm/llvm-project/issues/123569

In order to keep the patch at reasonable size, this PR only covers for
the llvm subproject, unittests excluded.

show more ...

[Target] Remove unused includes (NFC) (#116577)

Identified with misc-include-cleaner.

[NVPTX] add an optional early copy of byval arguments (#113384)

byval arguments in NVPTX are special. We're only allowed to read from
them using a special instruction, and if we ever need to write

[NVPTX] add an optional early copy of byval arguments (#113384)

byval arguments in NVPTX are special. We're only allowed to read from
them using a special instruction, and if we ever need to write to them
or take an address, we must make a local copy and use it, instead.

The problem is that local copies are very expensive, and we create them
very late in the compilation pipeline, so LLVM does not have much of a
chance to eliminate them, if they turn out to be unnecessary.

One way around that is to create such copies early on, and let them
percolate through the optimizations. The copying itself will never
trigger creation of another copy later on, as the reads are allowed. If
LLVM can eliminate it, it's a win. It the full optimization pipeline
can't remove the copy, that's as good as it gets in terms of the effort
we could've done, and it's certainly a much better effort than what we
do now.

This early injection of the copies has potential to create undesireable
side-effects, so it's disabled by default, for now, until it sees more
testing.

show more ...

[NVPTX] Allow MemTransferInst in adjustByValArgAlignment (#112462)

Before b7b28e770c46, AreSupportedUsers will skip
MemTransferInst, it may cause unexpected assertion.
https://godbolt.org/z/z5d691

[NVPTX] Allow MemTransferInst in adjustByValArgAlignment (#112462)

Before b7b28e770c46, AreSupportedUsers will skip
MemTransferInst, it may cause unexpected assertion.
https://godbolt.org/z/z5d691fj1
In b7b28e770c46, we start to allow MemTransferInst,
we should allow it in adjustByValArgAlignment too.

show more ...

[NVPTX] restrict `cvta.param` use to kernels only. (#112278)

If `cvta.param` is used in regular functions, it may produce an
invalid pointer. It's unclear if it's a bug in ptxas or we're not using

[NVPTX] restrict `cvta.param` use to kernels only. (#112278)

If `cvta.param` is used in regular functions, it may produce an
invalid pointer. It's unclear if it's a bug in ptxas or we're not using `cvta.param` correctly, but,
regardless of the underlying reason, the instruction has to be disabled for non-kernels, at least for now.

show more ...

[NVTPX] Copy kernel arguments as byte array (#110356)

Ensures that struct padding is not skipped, as it may contain actual
data if the struct is really a union.

The patch originated from a discu

[NVTPX] Copy kernel arguments as byte array (#110356)

Ensures that struct padding is not skipped, as it may contain actual
data if the struct is really a union.

The patch originated from a discussion on #53710

Fixes #53710

show more ...

[NFC] Rename `Intrinsic::getDeclaration` to `getOrInsertDeclaration` (#111752)

Rename the function to reflect its correct behavior and to be consistent
with `Module::getOrInsertFunction`. This is a

[NFC] Rename `Intrinsic::getDeclaration` to `getOrInsertDeclaration` (#111752)

Rename the function to reflect its correct behavior and to be consistent
with `Module::getOrInsertFunction`. This is also in preparation of
adding a new `Intrinsic::getDeclaration` that will have behavior similar
to `Module::getFunction` (i.e, just lookup, no creation).

show more ...

[NVPTX] Improve copy avoidance during lowering. (#106423)

On newer GPUs, where `cvta.param` instruction is available we can avoid
making byval arguments when their pointers are used in a few more c

[NVPTX] Improve copy avoidance during lowering. (#106423)

On newer GPUs, where `cvta.param` instruction is available we can avoid
making byval arguments when their pointers are used in a few more cases,
even when `__grid_constant__` is not specified.

- phi
- select
- memcpy from the parameter.

Switched pointer traversal from a DIY implementation to PtrUseVisitor.

show more ...

[DataLayout] Remove constructor accepting a pointer to Module (#102841)

The constructor initializes `*this` with `M->getDataLayout()`, which
is effectively the same as calling the copy constructor.

[DataLayout] Remove constructor accepting a pointer to Module (#102841)

The constructor initializes `*this` with `M->getDataLayout()`, which
is effectively the same as calling the copy constructor.
There does not seem to be a case where a copy would be necessary.

Pull Request: https://github.com/llvm/llvm-project/pull/102841

show more ...

[NVPTX] Improved support for grid_constant (#97112)

- Supports escaped grid_constant pointers less conservatively. Casts
uses inside Calls, PtrToInts, Stores where the pointer is a _value
operand_

[NVPTX] Improved support for grid_constant (#97112)

- Supports escaped grid_constant pointers less conservatively. Casts
uses inside Calls, PtrToInts, Stores where the pointer is a _value
operand_ to generic address space, immediately before the escape, while
keeping other uses in the param address space

- Related to: https://github.com/llvm/llvm-project/pull/96125

show more ...

[IR] Add getDataLayout() helpers to Function and GlobalValue (#96919)

Similar to https://github.com/llvm/llvm-project/pull/96902, this adds
`getDataLayout()` helpers to Function and GlobalValue, re

[IR] Add getDataLayout() helpers to Function and GlobalValue (#96919)

Similar to https://github.com/llvm/llvm-project/pull/96902, this adds
`getDataLayout()` helpers to Function and GlobalValue, replacing the
current `getParent()->getDataLayout()` pattern.

show more ...

[NVPTX] Basic support for "grid_constant" (#96125)

- Adds a helper function for checking whether an argument is a
[grid_constant](https://docs.nvidia.com/cuda/nvvm-ir-spec/index.html#supported-prop

[NVPTX] Basic support for "grid_constant" (#96125)

- Adds a helper function for checking whether an argument is a
[grid_constant](https://docs.nvidia.com/cuda/nvvm-ir-spec/index.html#supported-properties).
- Adds support for cvta.param using changes from
https://github.com/llvm/llvm-project/pull/95289
- Supports escaped grid_constant pointers conservatively, by casting all
uses to the generic address space with cvta.param.

show more ...

[NFC][RemoveDIs] Use iterators for insertion at various call-sites (#84736)

These are the last remaining "trivial" changes to passes that use
Instruction pointers for insertion. All of this should

[NFC][RemoveDIs] Use iterators for insertion at various call-sites (#84736)

These are the last remaining "trivial" changes to passes that use
Instruction pointers for insertion. All of this should be NFC, it's just
changing the spelling of how we identify a position.

In one or two locations, I'm also switching uses of getNextNode etc to
using std::next with iterators. This too should be NFC.

---------

Merged by: Stephen Tozer <stephen.tozer@sony.com>

show more ...

[llvm] Use range-based for loops (NFC)

[llvm] Remove uses of getWithSamePointeeType() (NFC)

[NVPTX] Infer AS of pointers passed to kernels as integers.

When pointers are passed within aggregates, we sometimes end up with IR that
loads them as integers and the converts them back to pointers

[NVPTX] Infer AS of pointers passed to kernels as integers.

When pointers are passed within aggregates, we sometimes end up with IR that
loads them as integers and the converts them back to pointers. Typically it's
due to a memcpy or SROA. E.g. https://godbolt.org/z/xM3n5daaa

Normally we treat all pointers passed to a CUDA kernel as global pointers and
the same treatment should be applied to the pointers we load/store as integers.

Differential Revision: https://reviews.llvm.org/D142664

show more ...

[NVPTX] Do not addrspacecast AS-specific kernel arguments.

Fixes https://github.com/llvm/llvm-project/issues/46954

The assumption that generic pointers passed to a CUDA kernel is CUDA-specific
and

[NVPTX] Do not addrspacecast AS-specific kernel arguments.

Fixes https://github.com/llvm/llvm-project/issues/46954

The assumption that generic pointers passed to a CUDA kernel is CUDA-specific
and should not be applied to non-CUDA compilations. Addrspacecasts to global AS
and back should never be applied to AS-specific pointers.

In order to make tests actually do the testing for non-CUDA compilation, we need
to get TargetMachine from the TargetPassConfig, instead of passing it explicitly
as a pass constructor argument.

Differential Revision: https://reviews.llvm.org/D142581

show more ...

[llvm] Use std::gcd (NFC)

This patch replaces calls to greatestCommonDivisor with std::gcd where
both arguments are known to be of unsigned. This means that
std::common_type_t of the two argument t

[llvm] Use std::gcd (NFC)

This patch replaces calls to greatestCommonDivisor with std::gcd where
both arguments are known to be of unsigned. This means that
std::common_type_t of the two argument types should just be the wider
one of the two.

show more ...

[llvm] Use value_or instead of getValueOr (NFC)

[llvm] Convert for_each to range-based for loops (NFC)

[NVPTX] Enhance vectorization of ld.param & st.param

Since function parameters and return values are passed via param space, we
can force special alignment for values hold in it which will add vecto

[NVPTX] Enhance vectorization of ld.param & st.param

Since function parameters and return values are passed via param space, we
can force special alignment for values hold in it which will add vectorization
options. This change may be done if the function has private or internal
linkage. Special alignment is forced during 2 phases.

1) Instruction selection lowering. Here we use special alignment for function
prototypes (changing both own return value and parameters alignment), call
lowering (changing both callee's return value and parameters alignment).

2) IR pass nvptx-lower-args. Here we change alignment of byval parameters that
belong to param space (or are casted to it). We only handle cases when all
uses of such parameters are loads from it. For such loads, we can change the
alignment according to special type alignment and the load offset. Then,
load-store-vectorizer IR pass will perform vectorization where alignment
allows it.

Special alignment calculated as maximum from default ABI type alignment and
alignment 16. Alignment 16 is chosen because it's the maximum size of
vectorized ld.param & st.param.

Before specifying such special alignment, we should check if it is a multiple
of the alignment that the type already has. For example, if a value has an
enforced alignment of 64, default ABI alignment of 4 and special alignment
of 16, we should preserve 64.

This patch will be followed by a refactoring patch that removes duplicating
code in handling byval and non-byval arguments.

Differential Revision: https://reviews.llvm.org/D120129

show more ...

Revert "[NVPTX] Enhance vectorization of ld.param & st.param"

This reverts commit f854434f0f2a01027bdaad8e6fdac5a782fce291.

Placed URL to wrong differential revision in commit message.

[NVPTX] Enhance vectorization of ld.param & st.param

Since function parameters and return values are passed via param space, we
can force special alignment for values hold in it which will add vecto

[NVPTX] Enhance vectorization of ld.param & st.param

Since function parameters and return values are passed via param space, we
can force special alignment for values hold in it which will add vectorization
options. This change may be done if the function has private or internal
linkage. Special alignment is forced during 2 phases.

1) Instruction selection lowering. Here we use special alignment for function
prototypes (changing both own return value and parameters alignment), call
lowering (changing both callee's return value and parameters alignment).

2) IR pass nvptx-lower-args. Here we change alignment of byval parameters that
belong to param space (or are casted to it). We only handle cases when all
uses of such parameters are loads from it. For such loads, we can change the
alignment according to special type alignment and the load offset. Then,
load-store-vectorizer IR pass will perform vectorization where alignment
allows it.

Special alignment calculated as maximum from default ABI type alignment and
alignment 16. Alignment 16 is chosen because it's the maximum size of
vectorized ld.param & st.param.

Before specifying such special alignment, we should check if it is a multiple
of the alignment that the type already has. For example, if a value has an
enforced alignment of 64, default ABI alignment of 4 and special alignment
of 16, we should preserve 64.

This patch will be followed by a refactoring patch that removes duplicating
code in handling byval and non-byval arguments.

Differential Revision: https://reviews.llvm.org/D121549

show more ...

[NVPTXLowerArgs] Use byval type

Instead of pointer element type.

[NFC] Remove uses of PointerType::getElementType()

Instead use either Type::getPointerElementType() or
Type::getNonOpaquePointerElementType().

This is part of D117885, in preparation for deprecatin

[NFC] Remove uses of PointerType::getElementType()

Instead use either Type::getPointerElementType() or
Type::getNonOpaquePointerElementType().

This is part of D117885, in preparation for deprecating the API.

show more ...