[IR] Increase max alignment to 4GB

Currently the max alignment representable is 1GB, see D108661.
Setting the align of an object to 4GB is desirable in some cases to make sure the lower 32 bits are

[IR] Increase max alignment to 4GB

Currently the max alignment representable is 1GB, see D108661.
Setting the align of an object to 4GB is desirable in some cases to make sure the lower 32 bits are clear which can be used for some optimizations, e.g. https://crbug.com/1016945.

This uses an extra bit in instructions that carry an alignment. We can store 15 bits of "free" information, and with this change some instructions (e.g. AtomicCmpXchgInst) use 14 bits.
We can increase the max alignment representable above 4GB (up to 2^62) since we're only using 33 of the 64 values, but I've just limited it to 4GB for now.

The one place we have to update the bitcode format is for the alloca instruction. It stores its alignment into 5 bits of a 32 bit bitfield. I've added another field which is 8 bits and should be future proof for a while. For backward compatibility, we check if the old field has a value and use that, otherwise use the new field.

Updating clang's max allowed alignment will come in a future patch.

Reviewed By: hans

Differential Revision: https://reviews.llvm.org/D110451

show more ...

[IR] Handle large element size when calculating GEP indices

This is a fix for the issue reported at
https://reviews.llvm.org/D110043#3019942:
The ElementSize is a uint64_t and as such may be larger

[IR] Handle large element size when calculating GEP indices

This is a fix for the issue reported at
https://reviews.llvm.org/D110043#3019942:
The ElementSize is a uint64_t and as such may be larger than the
index space, or be negative in the index space. This is UB, but
shouldn't cause assertion failures.

We address this by detecting whether the size is too large and
use a zero index in that case (which is always conservatively
correct).

Differential Revision: https://reviews.llvm.org/D110437

show more ...

[SystemZ][z/OS] Add GOFF Support to the DataLayout

- This patch adds in the GOFF mangling support to the LLVM data layout string. A corresponding additional line has been added into the data layout

[SystemZ][z/OS] Add GOFF Support to the DataLayout

- This patch adds in the GOFF mangling support to the LLVM data layout string. A corresponding additional line has been added into the data layout section in the language reference documentation.
- Furthermore, this patch also sets the right data layout string for the z/OS target in the SystemZ backend.

Reviewed By: uweigand, Kai, abhina.sreeskantharajan, MaskRay

Differential Revision: https://reviews.llvm.org/D109362

show more ...

[IR] Add helper to convert offset to GEP indices

We implement logic to convert a byte offset into a sequence of GEP
indices for that offset in a number of places. This patch adds a
DataLayout::getGE

[IR] Add helper to convert offset to GEP indices

We implement logic to convert a byte offset into a sequence of GEP
indices for that offset in a number of places. This patch adds a
DataLayout::getGEPIndicesForOffset() method, which implements the
core logic. I've updated SROA, ConstantFolding and InstCombine to
use it, and there's a few more places where it looks relevant.

Differential Revision: https://reviews.llvm.org/D110043

show more ...

Fix the default alignment of i1 vectors.

Currently, the default alignment is much larger than the actual size of
the vector in memory. Fix this to use a sane default.

For SVE, temporarily remove l

Fix the default alignment of i1 vectors.

Currently, the default alignment is much larger than the actual size of
the vector in memory. Fix this to use a sane default.

For SVE, temporarily remove lowering of load/store operations for
predicates with less than 16 elements. The layout the backend was
assuming for SVE predicates with less than 16 elements doesn't agree
with the frontend. More work probably needs to be done here.

This change is, strictly speaking, not backwards-compatible at the
bitcode level. But probably nobody is actually depending on that; i1
vectors in memory are rare, and the code that does use them probably
ends up forcing the alignment to something sane anyway. If we think
this is a concern, I can restrict this to scalable vectors for now
(where it's actually causing issues for me at the moment).

Differential Revision: https://reviews.llvm.org/D88994

show more ...

[StructLayout] Use TrailingObjects to allocate space for MemberOffsets.

MemberOffsets are stored at the end of StructLayout. The class
contains a single entry array to mark the start of the member
o

[StructLayout] Use TrailingObjects to allocate space for MemberOffsets.

MemberOffsets are stored at the end of StructLayout. The class
contains a single entry array to mark the start of the member
offsets. getStructLayout calculates the additional space needed
for additional elements before allocating memory.

This patch converts this to use TrailingObjects. This simplifies
the size computation in getStructLayout and gets rid of the
single entry array.

This is prep work, but to use TypeSize instead of uint64_t for
D98169. The single entry array doesn't work with TypeSize because
TypeSize doesn't have a default constructor. We thought this
change was an improvement by itself so we've separated it out.

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D99608

show more ...

[IR] Allow scalable vectors in structs to support intrinsics returning multiple values.

RISC-V would like to use a struct of scalable vectors to return multiple
values from intrinsics. This woud als

[IR] Allow scalable vectors in structs to support intrinsics returning multiple values.

RISC-V would like to use a struct of scalable vectors to return multiple
values from intrinsics. This woud also be needed for target independent
intrinsics like llvm.sadd.overflow.

This patch removes the existing restriction for this. I've modified
StructType::isSized to consider a struct containing scalable vectors
as unsized so the verifier won't allow loads/stores/allocas of these
structs.

Reviewed By: sdesmalen

Differential Revision: https://reviews.llvm.org/D94142

show more ...

[X86] Add x86_amx type for intel AMX.

The x86_amx is used for AMX intrisics. <256 x i32> is bitcast to x86_amx when
it is used by AMX intrinsics, and x86_amx is bitcast to <256 x i32> when it
is use

[X86] Add x86_amx type for intel AMX.

The x86_amx is used for AMX intrisics. <256 x i32> is bitcast to x86_amx when
it is used by AMX intrinsics, and x86_amx is bitcast to <256 x i32> when it
is used by load/store instruction. So amx intrinsics only operate on type x86_amx.
It can help to separate amx intrinsics from llvm IR instructions (+-*/).
Thank Craig for the idea. This patch depend on https://reviews.llvm.org/D87981.

Differential Revision: https://reviews.llvm.org/D91927

show more ...

[DL] Inline getAlignmentInfo() implementation (NFC)

Apart from getting the entry in the table (which is already a
separate function), the remaining logic is different for all
alignment types and is

[DL] Inline getAlignmentInfo() implementation (NFC)

Apart from getting the entry in the table (which is already a
separate function), the remaining logic is different for all
alignment types and is better combined with getAlignment().

This is a minor efficiency improvement, and should make further
improvements like using separate storage for different alignment
types simpler.

show more ...

[DL] Optimize address space zero lookup (NFC)

Information for pointer size/alignment/etc is queried a lot, but
the binary search based implementation makes this fairly slow.

Add an explicit check f

[DL] Optimize address space zero lookup (NFC)

Information for pointer size/alignment/etc is queried a lot, but
the binary search based implementation makes this fairly slow.

Add an explicit check for address space zero and skip the search
in that case -- we need to specially handle the zero address space
anyway, as it serves as the fallback for all address spaces that
were not explicitly defined.

I initially wanted to simply replace the binary search with a
linear search, which would handle both address space zero and the
general case efficiently, but I was not sure whether there are
any degenerate targets that use more than a handful of declared
address spaces (in-tree, even AMDGPU only declares six).

show more ...

Add a default address space for globals to DataLayout

This is similar to the existing alloca and program address spaces (D37052)
and should be used when creating/accessing global variables.
We need

Add a default address space for globals to DataLayout

This is similar to the existing alloca and program address spaces (D37052)
and should be used when creating/accessing global variables.
We need this in our CHERI fork of LLVM to place all globals in address space 200.
This ensures that values are accessed using CHERI load/store instructions
instead of the normal MIPS/RISC-V ones.

The problem this is trying to fix is that most of the time the type of
globals is created using a simple PointerType::getUnqual() (or ::get() with
the default address-space value of 0). This does not work for us and we get
assertion/compilation/instruction selection failures whenever a new call
is added that uses the default value of zero.

In our fork we have removed the default parameter value of zero for most
address space arguments and use DL.getProgramAddressSpace() or
DL.getGlobalsAddressSpace() whenever possible. If this change is accepted,
I will upstream follow-up patches to use DL.getGlobalsAddressSpace() instead
of relying on the default value of 0 for PointerType::get(), etc.

This patch and the follow-up changes will not have any functional changes
for existing backends with the default globals address space of zero.
A follow-up commit will change the default globals address space for
AMDGPU to 1.

Reviewed By: dylanmckay

Differential Revision: https://reviews.llvm.org/D70947

show more ...

[SVE] Make ElementCount members private

This patch changes ElementCount so that the Min and Scalable
members are now private and can only be accessed via the get
functions getKnownMinValue() and isS

[SVE] Make ElementCount members private

This patch changes ElementCount so that the Min and Scalable
members are now private and can only be accessed via the get
functions getKnownMinValue() and isScalable(). In addition I've
added some other member functions for more commonly used operations.
Hopefully this makes the class more useful and will reduce the
need for calling getKnownMinValue().

Differential Revision: https://reviews.llvm.org/D86065

show more ...

[llvm] support graceful failure of DataLayout parsing

Existing implementation always aborts on syntax errors in a DataLayout
description. While this is meaningful for consuming textual IR modules, i

[llvm] support graceful failure of DataLayout parsing

Existing implementation always aborts on syntax errors in a DataLayout
description. While this is meaningful for consuming textual IR modules, it is
inconvenient for users that may need fine-grained control over the layout from,
e.g., command-line options. Propagate errors through the parsing functions and
only abort in the top-level parsing function instead.

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D85650

show more ...

[XCOFF][AIX] Use 'L..' instead of '.L' for getPrivateGlobalPrefix in DataLayout

Summary:
D80831 changed part of the prefix usage for AIX.
But there are other places getting prefix from DataLayout.
T

[XCOFF][AIX] Use 'L..' instead of '.L' for getPrivateGlobalPrefix in DataLayout

Summary:
D80831 changed part of the prefix usage for AIX.
But there are other places getting prefix from DataLayout.
This patch intends to make prefix usage consistent on AIX.

Reviewed by: hubert.reinterpretcast, daltenty

Differential Revision: https://reviews.llvm.org/D81270

show more ...

[Alignment][NFC] Transition and simplify calls to DL::getABITypeAlignment

This patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/

[Alignment][NFC] Transition and simplify calls to DL::getABITypeAlignment

This patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html
See this patch for the introduction of the type: https://reviews.llvm.org/D64790

Differential Revision: https://reviews.llvm.org/D82956

show more ...

[Alignment][NFC] migrate DataLayout::getPreferredAlignment

This patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-J

[Alignment][NFC] migrate DataLayout::getPreferredAlignment

This patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html
See this patch for the introduction of the type: https://reviews.llvm.org/D64790

Differential Revision: https://reviews.llvm.org/D82752

show more ...

[SVE] Fix scalable vector bug in DataLayout::getIntPtrType

Fixed an issue in DataLayout::getIntPtrType where we were assuming
the input type was always a fixed vector type, which isn't true.

Added

[SVE] Fix scalable vector bug in DataLayout::getIntPtrType

Fixed an issue in DataLayout::getIntPtrType where we were assuming
the input type was always a fixed vector type, which isn't true.

Added a test that exposed the problem to:

Transforms/InstCombine/vector_gep1.ll

Differential Revision: https://reviews.llvm.org/D82294

show more ...

Remove references to the 4.0 release as a major breaking (NFC)

This is cleaning up comments (mostly in the bitcode handling) about
removing some backward compatibility aspect in the 4.0 release.
His

Remove references to the 4.0 release as a major breaking (NFC)

This is cleaning up comments (mostly in the bitcode handling) about
removing some backward compatibility aspect in the 4.0 release.
Historically, "4.0" was used during the development of the 3.x
versions as "this future major breaking change version". At the time
the major number was used to indicate the compatibility. When we
reached 3.9 we decided to change the numbering, instead of going to
3.10 we went to 4.0 but after changing the meaning of the major
number to not mean anything anymore with respect to bitcode backward
compatibility.

The current policy
(https://llvm.org/docs/DeveloperPolicy.html#ir-backwards-compatibility)
indicates only now:

The current LLVM version supports loading any bitcode since version 3.0.

Differential Revision: https://reviews.llvm.org/D82514

show more ...

[IR] Prefer scalar type for struct indexes in GEP constant expressions.

This has two advantages: one, it's simpler, and two, it doesn't require
heroic pattern matching with scalable vectors.

Also i

[IR] Prefer scalar type for struct indexes in GEP constant expressions.

This has two advantages: one, it's simpler, and two, it doesn't require
heroic pattern matching with scalable vectors.

Also includes a small fix to DataLayout to allow the scalable vector
testcase to work correctly.

Differential Revision: https://reviews.llvm.org/D82061

show more ...

[SVE] Eliminate calls to default-false VectorType::get() from IR

Reviewers: efriedma, kmclaughlin, sdesmalen, dexonsmith, dblaikie

Reviewed By: efriedma

Subscribers: tschuett, hiraditya, rkruppe,

[SVE] Eliminate calls to default-false VectorType::get() from IR

Reviewers: efriedma, kmclaughlin, sdesmalen, dexonsmith, dblaikie

Reviewed By: efriedma

Subscribers: tschuett, hiraditya, rkruppe, psnobl, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D80261

show more ...

[IR][BFloat] Add BFloat IR type

Summary:
The BFloat IR type is introduced to provide support for, initially, the BFloat16
datatype introduced with the Armv8.6 architecture (optional from Armv8.2
onw

[IR][BFloat] Add BFloat IR type

Summary:
The BFloat IR type is introduced to provide support for, initially, the BFloat16
datatype introduced with the Armv8.6 architecture (optional from Armv8.2
onwards). It has an 8-bit exponent and a 7-bit mantissa and behaves like an IEEE
754 floating point IR type.

This is part of a patch series upstreaming Armv8.6 features. Subsequent patches
will upstream intrinsics support and C-lang support for BFloat.

Reviewers: SjoerdMeijer, rjmccall, rsmith, liutianle, RKSimon, craig.topper, jfb, LukeGeeson, sdesmalen, deadalnix, ctetreau

Subscribers: hiraditya, llvm-commits, danielkiss, arphaman, kristof.beyls, dexonsmith

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D78190

show more ...

[SVE] Fix getAlignmentInfo for scalable vectors

When calculating the natural alignment for scalable vectors it
is acceptable to calculate an allocation size based on the minimum
number of elements i

[SVE] Fix getAlignmentInfo for scalable vectors

When calculating the natural alignment for scalable vectors it
is acceptable to calculate an allocation size based on the minimum
number of elements in the vector.

This code path is exercised by an existing test:

CodeGen/AArch64/sve-intrinsics-int-arith.ll

Differential Revision: https://reviews.llvm.org/D79475

show more ...

[SVE] Add new VectorType subclasses

Summary:
Introduce new types for fixed width and scalable vectors.

Does not remove getNumElements yet so as to not break code during transition
period.

Reviewer

[SVE] Add new VectorType subclasses

Summary:
Introduce new types for fixed width and scalable vectors.

Does not remove getNumElements yet so as to not break code during transition
period.

Reviewers: deadalnix, efriedma, sdesmalen, craig.topper, huntergr

Reviewed By: sdesmalen

Subscribers: jholewinski, arsenm, jvesely, nhaehnle, mehdi_amini, rriddle, jpienaar, burmako, shauheen, antiagainst, nicolasvasilache, csigg, arpith-jacob, mgester, lucyrfox, liufengdb, kerbowa, Joonsoo, grosul1, frgossen, lldb-commits, tschuett, hiraditya, rkruppe, psnobl, llvm-commits

Tags: #llvm, #lldb

Differential Revision: https://reviews.llvm.org/D77587

show more ...

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly m

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.

show more ...

[Alignment][NFC] Deprecate Align::None()

Summary:
This is a follow up on https://reviews.llvm.org/D71473#inline-647262.
There's a caveat here that `Align(1)` relies on the compiler understanding of

[Alignment][NFC] Deprecate Align::None()

Summary:
This is a follow up on https://reviews.llvm.org/D71473#inline-647262.
There's a caveat here that `Align(1)` relies on the compiler understanding of `Log2_64` implementation to produce good code. One could use `Align()` as a replacement but I believe it is less clear that the alignment is one in that case.

Reviewers: xbolva00, courbet, bollu

Subscribers: arsenm, dylanmckay, sdardis, nemanjai, jvesely, nhaehnle, hiraditya, kbarton, jrtc27, atanasyan, jsji, Jim, kerbowa, cfe-commits, llvm-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D73099

show more ...