libc - pthread_key_create() adjustments

* Have libc's stub conditionals ignore any error return from
pthread_key_create() stubs, in case we decide to change the
stub in the future.

* Change pth

libc - pthread_key_create() adjustments

* Have libc's stub conditionals ignore any error return from
pthread_key_create() stubs, in case we decide to change the
stub in the future.

* Change pthread_key_create() to start allocating keys at key #1
to work around issues with third party programs making assumptions
about the valid key range.

show more ...

libc - Fix bug in recent malloc_usable_size() support

* Add missing unlock in the bigalloc check path

* Fixes miniruby deadlock and other threaded uses of malloc_usable_size()
on large memory blo

libc - Fix bug in recent malloc_usable_size() support

* Add missing unlock in the bigalloc check path

* Fixes miniruby deadlock and other threaded uses of malloc_usable_size()
on large memory blocks.

show more ...

libc: Add malloc_usable_size(3) support.

Submitted-by: @dillon

libc - Fix a serious memory leak in the nmalloc code

* Fix a bug where mtmagazine_free() would assign mt->newmag without
re-checking its contents, potentially overwriting an assignment
made by _

libc - Fix a serious memory leak in the nmalloc code

* Fix a bug where mtmagazine_free() would assign mt->newmag without
re-checking its contents, potentially overwriting an assignment
made by _slaballoc(), causing a memory leak.

* This bug generally rears its ugly head in heavily pthreaded programs,
but it can occur under other conditions too.

Reported-by: hsw (irc)

show more ...

libc - nmalloc clean part 3

* Cleanup the sigblockall()/sigunblockall() use cases to
remove unnecessary recursive counts. It suffices to wrap
the exported functions, fork callbacks, and the des

libc - nmalloc clean part 3

* Cleanup the sigblockall()/sigunblockall() use cases to
remove unnecessary recursive counts. It suffices to wrap
the exported functions, fork callbacks, and the destructor.

show more ...

libc - nmalloc clean part 2

* Clean up a lot of unused #define's and other constants.

* Increase the maximum capacity of a magazine to roughly
512 (M_MAX_ROUNDS). Continue to use a fixed structu

libc - nmalloc clean part 2

* Clean up a lot of unused #define's and other constants.

* Increase the maximum capacity of a magazine to roughly
512 (M_MAX_ROUNDS). Continue to use a fixed structure.

* Set the magazine capacity dynamically based on the allocation
size. Magazines for smaller allocations have larger capacities.
This should improve performance for tiny allocations.

show more ...

libc - nmalloc clean part 1

* Add an __isexiting global that the malloc thread destructor
can use to detect program exit (verses just thread exit),
allowing it to avoid burning unnecessary cpu.

libc - nmalloc clean part 1

* Add an __isexiting global that the malloc thread destructor
can use to detect program exit (verses just thread exit),
allowing it to avoid burning unnecessary cpu. That is, it
doesn't bother to clean stuff if the program is on its way
out.

* Fresh allocations when the magazines are empty no longer
dive into the global structure (with accompanying locks)
for every allocation and will instead burst a bunch of chunks
into the local mtmagazine to reduce lock contention.

* Reduce the time the global slgd_lock() is held by rejiggering
the code a bit.

* Reduce overhead for allocating the magazine structure by
allowing the thead-local magazine to be accessed if it
happens to contain approriate free chunks.

We no longer disable the thread-local mtmagazine during
the recursive _slaballoc() call and instead pass a flag
to prevent an infinite recursion.

* Fix the confusing slgd_unlock() call that was previously in
zone_free(). slgd_unlock() is now called prior to the
zone_free().

show more ...

libc - Implement sigblockall() and sigunblockall()

* Signal safety is becoming a defacto requirement for most of libc and
pthreads. In particular, the memory allocator. Given the chances of
te

libc - Implement sigblockall() and sigunblockall()

* Signal safety is becoming a defacto requirement for most of libc and
pthreads. In particular, the memory allocator. Given the chances of
teaching tens of thousands of programmers about signal safety, and
just making it work in libc and pthreads, only one of these two
possibilities is actually realizable.

In particular, high-level languages have become so complex, and some
applications (chrome, firefox, etc) have become so complex, that the
code is regularly tripping over signal safety issues.

However, implementing signal safety with current mechanisms is extremely
expensive due to the need for multiple system calls. To whit,
DragonFlyBSD now has a mechanism that does not require system calls
in the critical path.

* Implement sigblockall() and sigunblockall(). These functions
leverage the new /dev/lpmap per-thread shared page mechanism
to provide a way to temporary block the dispatch of all maskable
signals without having to make any system calls.

These are extremely fast routines.

- Reentrant / Recursable

- Temporarily blocks any dispatch of a maskable asynchronous signal
to the calling thread. Other threads are not affected... this is
a per-thread mechanism.

- The last sigunblockall() will immediately dispatch any blocked
signals.

- The normal signal mask is not affected by these routines.

- Does not block signals caused by synchronous traps.

- The current recursion count is retained on [v]fork() to ease
coding and to also allow signals to be temporarily blocked across a
fork until the child process is ready to deal with them, if desired.

* Implement signal safety for most of pthreads. All temporary internal
mutexes are now wrapped with sigblockall() and sigunblockall().

* Implement signal safety for the malloc subsystem. All functions
are wrawpped with sigblockall() and sigunblockall().

These implementations make lang/mono and lang/rust far more reliable
than they were before. Where 9 out of 10 builds used to fail, now
they succeed.

show more ...

Cleanup <sys/uio.h> includes.

No longer needed after <sys/ktrace.h> cleanup.

nmalloc(3): Add missing #include "namespace.h" and adjust.

libc: Add prototypes for various functions we use in __weak_reference().

libc/stdlib: Fix two -Wunused-parameter warnings.

libc/stdlib: Refactor aligned_alloc().

* Lift the restriction from posix_memalign() that alignment should be
more or equal to sizeof(void *).
* Make aligned_alloc() a weak symbol.
* Adjust

libc/stdlib: Refactor aligned_alloc().

* Lift the restriction from posix_memalign() that alignment should be
more or equal to sizeof(void *).
* Make aligned_alloc() a weak symbol.
* Adjust the manpage.

show more ...

nmalloc - Try bigcache in posix_memalign() for big PAGE_SIZE aligned allocs.

* This improves performance (by reducing the amount of mmap() calls required
and avoiding some amount of page faults) f

nmalloc - Try bigcache in posix_memalign() for big PAGE_SIZE aligned allocs.

* This improves performance (by reducing the amount of mmap() calls required
and avoiding some amount of page faults) for applications which frequently
call posix_memalign() with a 4096 byte alignment for allocations bigger
than 8KB. Allocations of 8KB and smaller are efficiently handled by the
SLAB allocator already.

show more ...

nmalloc - Cache 4K and 8K allocations in posix_memalign()

* Adjust posix_memalign() to use the slab allocator for 4K and 8K
allocations same as malloc() does, to improve performance for
programs

nmalloc - Cache 4K and 8K allocations in posix_memalign()

* Adjust posix_memalign() to use the slab allocator for 4K and 8K
allocations same as malloc() does, to improve performance for
programs using posix_memalign().

show more ...

libc/nmalloc: Handle size_t overflows in malloc().

Similar to e9586122ca8be1e743aa9cd2f9622dfe2f434ece.

libc/nmalloc: Handle size_t overflows in posix_memalign().

Some algorithms/programs try to heuristically deduce alignment rules.
Instead of confusing them with minimal allocations for size == (size_

libc/nmalloc: Handle size_t overflows in posix_memalign().

Some algorithms/programs try to heuristically deduce alignment rules.
Instead of confusing them with minimal allocations for size == (size_t)-7 and
friends at different alignment values, just return ENOMEM since POSIX does not
explictly specify how alignment overflows should behave. Programs should keep
track of pointers they try to allocate (including size == 0 case too).

show more ...

libc/nmalloc: Whistespace cleanup.

No functional change.

libc and pthreads - Fix atfork issues with nmalloc, update dmalloc

* Implement atfork handling for nmalloc. As part of this, refactor
some of nmalloc.

* Remove ZERO_LENGTH_PTR from nmalloc. Ins

libc and pthreads - Fix atfork issues with nmalloc, update dmalloc

* Implement atfork handling for nmalloc. As part of this, refactor
some of nmalloc.

* Remove ZERO_LENGTH_PTR from nmalloc. Instead, force 0-byte
allocations to allocate 1 byte. The standard requires unique
pointers to be returned.

* For now go back to a single depot lock instead of a per-zone
lock. It is unclear whether multi-threaded performance will
suffer or not, but its the only way to implement atfork handling.

* Implement proper atfork interlocks for nmalloc via pthreads to avoid
corruption when heavily threaded programs call fork().

* Bring dmalloc up to date in various ways, including properly
implementing a minimum 16-byte alignment for allocations >= 16 bytes,
and atfork handling. Also use a global depot lock for the same
reason we use it in nmalloc, and implement a front-end magazine
shortcut for any allocations <= 2MB.

Reported-by: mneumann

show more ...

Define __* symbols for the malloc(3) functions to make wrapping easier.

This should match what FreeBSD and other systems do. Where the __* variants
always refer to the libc function, but the actual

Define __* symbols for the malloc(3) functions to make wrapping easier.

This should match what FreeBSD and other systems do. Where the __* variants
always refer to the libc function, but the actual malloc(), free() etc.
symbols can be overrided by a malloc wrapper.

show more ...

libc/nmalloc: Remove some kernel code remains.

libc - Fix malloc() alignment for small allocations

* malloc()'s slab allocator was set to use 8-byte alignment
for any allocation < 128 bytes that was not otherwise on
an integral alignment bou

libc - Fix malloc() alignment for small allocations

* malloc()'s slab allocator was set to use 8-byte alignment
for any allocation < 128 bytes that was not otherwise on
an integral alignment boundary. This breaks GCC-7 which assumes
16-byte alignment for non-16-integral sizes < 128 bytes. e.g.
if 18 bytes is allocated, GCC-7 assumes the resulting pointer will
be 16-byte-aligned.

* The standard is somewhat deficient in its characterization of what the
required alignment should be, because there are already instructions
which prefer 32 and 64 byte alignments, but are relaxed on Intel to
only require 16-byte alignments (aka %ymm and %zmm registers in the
vector extensions), and its stupid to enforce even larger alignments
for tiny allocations.

* But generally speaking it makes sense to enforce a 16-byte alignment
for any allocations >= 16 bytes, regardless of the size being passed-in
not being 16-byte aligned, and this change does that. Allocations of
less than 16 bytes will still be 8-byte aligned because it is phenominally
wasteful for them not to be.

Reported-by: marino

show more ...

libc: Do not set errno when mmap(MAP_TRYFIXED) fails.

If this mmap call fails, we'll fall back to moving the allocation to
a different virtual address. Setting errno here might result in
functions f

libc: Do not set errno when mmap(MAP_TRYFIXED) fails.

If this mmap call fails, we'll fall back to moving the allocation to
a different virtual address. Setting errno here might result in
functions further up the call stack misinterpreting an error condition.

Indeed, this surfaced in git's calling of getdelim() in a loop, where
getdelim() would return -1 at EOF, but the caller would find ENOMEM in
errno from a previous, successful, call and interpret that as an OOM.

Fix-to: 07a8ffeaab3fc830679c6836a6ee198ad1a3275f

Suggested-by: aggelos

show more ...

libc - Improve realloc() performance.

* Oversized free()'s <= 1MB will be cached in a 16-entry array. Any oversized
allocation can make use of the cache, even if available cache entries are
sig

libc - Improve realloc() performance.

* Oversized free()'s <= 1MB will be cached in a 16-entry array. Any oversized
allocation can make use of the cache, even if available cache entries are
significantly larger than desired, in order to accomodate application
buffer-growing behavior. This means that applications which realloc() a
lot (at least as long as the buffer is <= 1MB) will wind up being handed
a ~1MB buffer quite frequently, eliminating new mmap() operations.

* Keeps track of the excess in oversized allocations and will garbage-collect
it if it grows too large (typically > 16MB). This is primarily what allows
us to be generous.

* realloc() will attempt to use MAP_TRYFIXED to extend an oversized allocation
before giving up and remapping it.

* Improves programs which realloc() or malloc()/free() large numbers of
oversized blocks by reducing or eliminating mmap()/munmap() calls.

show more ...

nmalloc - Improve incremental reallocs()

* Fix realloc() issue that caused unnecessary unmap/remap operations
due to the page coloring optimization.

* In addition, scale large reallocations by al

nmalloc - Improve incremental reallocs()

* Fix realloc() issue that caused unnecessary unmap/remap operations
due to the page coloring optimization.

* In addition, scale large reallocations by allowing more wiggle room
as the allocation becomes larger so overhead isn't quite as geometrically
bad for programs which do incremental realloc()s to huge sizes.

Initial-patch-by: vadaszi Bug #2832

show more ...