[scudo] Calling initCache() in init() of SizeClassAllocatorLocalCache (#71427)

initCacheMaybe() will init all the size class arrays at once and it
doesn't have much work to do even if it supports p

[scudo] Calling initCache() in init() of SizeClassAllocatorLocalCache (#71427)

initCacheMaybe() will init all the size class arrays at once and it
doesn't have much work to do even if it supports partial initialization.
This avoids the call to initCacheMaybe in each allocate()/deallocate().

show more ...

[scudo] Pass the max number of blocks to popBlocks (#70243)

Make the cache have the fully control on how many blocks to be popped
(At before, it depended the number of blocks stored in the
Transfe

[scudo] Pass the max number of blocks to popBlocks (#70243)

Make the cache have the fully control on how many blocks to be popped
(At before, it depended the number of blocks stored in the
TransferBatch)

show more ...

Reapply "[scudo] Make local cache be agnostic to the type of node in … (#68633)

…f… (#68626)

This reverts commit 8dd9615dfbd148993964ea6f2de6c6a94f49660c.

1. Fixed the ambiguous aliasing
2. F

Reapply "[scudo] Make local cache be agnostic to the type of node in … (#68633)

…f… (#68626)

This reverts commit 8dd9615dfbd148993964ea6f2de6c6a94f49660c.

1. Fixed the ambiguous aliasing
2. Fixed the type conversion warning

show more ...

Revert "[scudo] Make local cache be agnostic to the type of node in f… (#68626)

…reelist (#67379)"

This reverts commit b9c6737ba7307308ecb6ec4cecc4c07e48e7c141.

[scudo] Make local cache be agnostic to the type of node in freelist (#67379)

This change moves the `TransferBatch` and `BatchGroup` out of
SizeClassAllocatorLocalCache. It allows us that the node

[scudo] Make local cache be agnostic to the type of node in freelist (#67379)

This change moves the `TransferBatch` and `BatchGroup` out of
SizeClassAllocatorLocalCache. It allows us that the node in freelist can
store more blocks instead of depending on the number of blocks cached.

That means we will be able to store more blocks in each node of freelist
and therefore reduce the memory used by BatchClass (with little
performance overhead). Note that we haven't enabled that in this patch.
This is the first step of this transition.

show more ...

Reland "[scudo] Support partial concurrent page release in SizeClassAllocator64"

This reverts commit 2f04b688aadef747172a8f4a7d1658dc049b1e24.

Reviewed By: cferris

Differential Revision: https://r

Reland "[scudo] Support partial concurrent page release in SizeClassAllocator64"

This reverts commit 2f04b688aadef747172a8f4a7d1658dc049b1e24.

Reviewed By: cferris

Differential Revision: https://reviews.llvm.org/D155015

show more ...

[scudo] Try to release pages after unlocking the TSDs

This increases the parallelism and the usage of TSDs

Reviewed By: cferris

Differential Revision: https://reviews.llvm.org/D152988

[scudo] PopBatch after populateFreeList()

Ensure the thread that refills freelist will get the Batch without
contending the lock in SizeClassAllocator64.

Reviewed By: cferris

Differential Revision

[scudo] PopBatch after populateFreeList()

Ensure the thread that refills freelist will get the Batch without
contending the lock in SizeClassAllocator64.

Reviewed By: cferris

Differential Revision: https://reviews.llvm.org/D152419

show more ...

[scudo] Support printing the status of cached blocks

The output looks like,
Stats: SharedTSDs: 2 available; total 8
Shared TSD[0]:
00 ( 64): cached: 10 max: 26
32 ( 65552):

[scudo] Support printing the status of cached blocks

The output looks like,
Stats: SharedTSDs: 2 available; total 8
Shared TSD[0]:
00 ( 64): cached: 10 max: 26
32 ( 65552): cached: 1 max: 2
Shared TSD[1]:
No block is cached.

Differential Revision: https://reviews.llvm.org/D148755

show more ...

[scudo] Use bytes-in-freelist as a hint of page release

Tracking the pushed bytes between to releaseToOSMaybe calls may lead to
a overestimated case that if we do malloc 2KB -> free 2KB -> malloc 2K

[scudo] Use bytes-in-freelist as a hint of page release

Tracking the pushed bytes between to releaseToOSMaybe calls may lead to
a overestimated case that if we do malloc 2KB -> free 2KB -> malloc 2KB
-> free 2KB, we may think we have released 4KB but it only releases 2KB
actually. Switch to use bytes-in-freelist excludes more cases that can't
release the pages

Reviewed By: cferris

Differential Revision: https://reviews.llvm.org/D146400

show more ...

[scudo] Make the boundary of memory group aligned with region begin

This alignment guarantee enables simpler group range check while page
releasing and a potential optimization which is, now all the

[scudo] Make the boundary of memory group aligned with region begin

This alignment guarantee enables simpler group range check while page
releasing and a potential optimization which is, now all the pointers
from the same group are also inth same region, that means the complexity
in markFreeBlocks() can be reduced as well.

Reviewed By: cferris

Differential Revision: https://reviews.llvm.org/D142931

show more ...

Revert "Revert "[scudo] Support partial page releasing""

This reverts commit 1cf1b36112b3d475d443909881b02799b548e6b8.

Reviewed By: cferris

Differential Revision: https://reviews.llvm.org/D136027

Reland "[scudo] Manage free blocks in BatchGroup."

This is not a pure revert of c929bcb7d85700494217f3a2148549f8757e0eed.
It also includes a bug fix.

Differential Revision: https://reviews.llvm.org

Reland "[scudo] Manage free blocks in BatchGroup."

This is not a pure revert of c929bcb7d85700494217f3a2148549f8757e0eed.
It also includes a bug fix.

Differential Revision: https://reviews.llvm.org/D136029

show more ...

[scudo] Fix implicitly narrow casting (NFC)

u16 may be promoted to int by arithmetic type conversion. Do an explicit
cast to avoid certain compiler's warning. This fixes the problem
introduced by 0f

[scudo] Fix implicitly narrow casting (NFC)

u16 may be promoted to int by arithmetic type conversion. Do an explicit
cast to avoid certain compiler's warning. This fixes the problem
introduced by 0fb2aeef5310eaba2915b30810464a744a80da15

Differential Revision: https://reviews.llvm.org/D135985

show more ...

Revert "[scudo] Manage free blocks in BatchGroup."

This reverts commit cf9d7f55d3bec7640fa8b2f8ec1d9c1268233caa.

Revert "[scudo] Support partial page releasing"

This reverts commit 9c26f51f5e178ac0fda98419e3a61d205d3b58b1.

Revert "[scudo] Fix implicitly narrow casting (NFC)"

This reverts commit fd7c7ad4fe0138314b922ea0db1691d5a679cc75.

[scudo] Fix implicitly narrow casting (NFC)

u16 may be promoted to int by arithmetic type conversion. Do an explicit
cast to avoid certain compiler's warning.

Differential Revision: https://reviews

[scudo] Fix implicitly narrow casting (NFC)

u16 may be promoted to int by arithmetic type conversion. Do an explicit
cast to avoid certain compiler's warning.

Differential Revision: https://reviews.llvm.org/D135945

show more ...

[scudo] Support partial page releasing

Block grouping enables us doing partial page releasing so that we can
release the pages in a finer granularity. Which means we don't need to
visit all blocks t

[scudo] Support partial page releasing

Block grouping enables us doing partial page releasing so that we can
release the pages in a finer granularity. Which means we don't need to
visit all blocks to determine which pages are unused. Besides, this
means we can do incremental page releasing depends on the number fo free
blocks.

Reviewed By: cryptoad, cferris

Differential Revision: https://reviews.llvm.org/D134226

show more ...

[scudo] Manage free blocks in BatchGroup.

Scudo is supposed to allocate any blocks across the entired mapped
pages and each page is equally likely to be selected. Which means Scudo
is leaning to tou

[scudo] Manage free blocks in BatchGroup.

Scudo is supposed to allocate any blocks across the entired mapped
pages and each page is equally likely to be selected. Which means Scudo
is leaning to touch as many pages as possible. This brings better
security but it also sacrifices the chance of releasing dirty pages.

To alleviate the unmanagable footprint growing, this CL introduces the
BatchGroup concept. Each blocks will be classified into a BatchGroup
according to its address. While allocation, we are leaning to allocate
blocks in the same group first. Note that the blocks selected from a
group is still random over several pages. At the same time, we have
better prediction of dirty page growing speed. Besides, we are able to
do partial page releasing by examing part of BatchGroups.

Reviewed By: cryptoad, cferris

Differential Revision: https://reviews.llvm.org/D133897

show more ...

Use u16 to store Count/MaxCount

The Count/MaxCount used in TransferBatch and PerClass can be fit in u16 in
current configurations and it's also reasonable to have a u16 limit. The
spare 16 bits will

Use u16 to store Count/MaxCount

The Count/MaxCount used in TransferBatch and PerClass can be fit in u16 in
current configurations and it's also reasonable to have a u16 limit. The
spare 16 bits will be used for additional status like pages mapping
status in a TransferBatch.

Reviewed By: cryptoad, cferris, vitalybuka

Differential Revision: https://reviews.llvm.org/D133145

show more ...

[scudo] Get rid of initLinkerInitialized

Now that everything is forcibly linker initialized, it feels like a
good time to get rid of the `init`/`initLinkerInitialized` split.

This allows to get rid

[scudo] Get rid of initLinkerInitialized

Now that everything is forcibly linker initialized, it feels like a
good time to get rid of the `init`/`initLinkerInitialized` split.

This allows to get rid of various `memset` construct in `init` that
gcc complains about (this fixes a Fuchsia open issue).

I added various `DCHECK`s to ensure that we would get a zero-inited
object when entering `init`, which required ensuring that
`unmapTestOnly` leaves the object in a good state (tests are currently
the only location where an allocator can be "de-initialized").

Running the tests with `--gtest_repeat=` showed no issue.

Differential Revision: https://reviews.llvm.org/D103119

show more ...

[scudo] Don't track free/use stats for transfer batches.

The Scudo C unit tests are currently non-hermetic. In particular, adding
or removing a transfer batch is a global state of the allocator that

[scudo] Don't track free/use stats for transfer batches.

The Scudo C unit tests are currently non-hermetic. In particular, adding
or removing a transfer batch is a global state of the allocator that
persists between tests. This can cause flakiness in
ScudoWrappersCTest.MallInfo, because the creation or teardown of a batch
causes mallinfo's uordblks or fordblks to move up or down by the size of
a transfer batch on malloc/free.

It's my opinion that uordblks and fordblks should track the statistics
related to the user's malloc() and free() usage, and not the state of
the internal allocator structures. Thus, excluding the transfer batches
from stat collection does the trick and makes these tests pass.

Repro instructions of the bug:
1. ninja ./projects/compiler-rt/lib/scudo/standalone/tests/ScudoCUnitTest-x86_64-Test
2. ./projects/compiler-rt/lib/scudo/standalone/tests/ScudoCUnitTest-x86_64-Test --gtest_filter=ScudoWrappersCTest.MallInfo

Reviewed By: cryptoad

Differential Revision: https://reviews.llvm.org/D101653

show more ...

[scudo] Use require_constant_initialization

Attribute guaranties safe static initialization of globals.

Reviewed By: hctim

Differential Revision: https://reviews.llvm.org/D101514

Revert "[scudo] Use require_constant_initialization"

This reverts commit 7ad4dee3e733d820115f44cecce73ceb64c76450.