indent.

Remove leftovers of loaned pages.

ok kettenis@

The pmap_enter(9) function may fail even if we have enough free physical
memory. This happens if we can't allocate KVA to map that memory. This
is especially likely to happen with the arm64 pmap (a

The pmap_enter(9) function may fail even if we have enough free physical
memory. This happens if we can't allocate KVA to map that memory. This
is especially likely to happen with the arm64 pmap (and the riscv64 pmap
that is derived from it) since lock contention on the kernel map will
make us fail to allocate the required KVA. But this may happen on other
architectures as well, especially those that don't define
__HAVE_PMAP_DIRECT.

Fix this issue by introducing a new pmap_populate() interface that may be
called to populate the page tables such that a subsequent pmap_enter(9)
call that uses the same virtual address will succeed. Use this in the
uvm fault handler after a failed pmap_enter(9) call before we retry the
fault.

tested by phessler@, mglocker@
ok mpi@

show more ...

Unlock underlying `uobj' when OOM in uvmfault_promote().

Found the hardway by sthen@, ok kettenis@

Move pmap_page_protect(PROT_NONE) call inside uvm_pagedeactivate().

ok tb@, kettenis@

Use a variable to represent the type of lock necessary for the lower fault.

For the moment an exclusive (RW_WRITE) lock is always used.

ok kettenis@, tb@

Read entry's values before releasing locks in uvm_fault_lower_io().

ok tb@, miod@, kettenis@

Merge identical code paths to promote data to a new anon into a new function.

ok tb@, miod@

Do not busy pages that are resident & fetched with PGO_LOCKED.

This is safe because the rwlock of the related object is never released until
the handler is done with the pages.

ok kettenis@, tb@

Return errno values rather than dying VM_PAGER_* in the pgo_fault() interfaces.

This doesn't introduce any value change. All errors are converted to EACCES
even if many could use EIO.

ok tb@, jsg@

Document that the original page during a CoW can be unlocked earlier.

ok tb@

Pass the rw_enter(9) type to amap_lock() in preparation for using shared locks.

ok tb@

Add missing wakeup & cleanup in error path.

ok tb@

Use uvm_pagewait() rather than re-rolling it.

ok miod@, tb@

Also call pmap_extract() before entering a page ahead for lower layer faults.

As for the upper layer, call pmap_update() only if, at least, a page has been
entered.

ok tb@, kettenis@

When paging ahead, delay calling pmap_extract() after checking for a valid page.

While here call pmap_update() only if, at least, a page has been entered.

ok tb@, kettenis@

Neighbor (fault ahead) pages are never mapped with the wired attribute.

Wired faults are always "narrow". That means the fault handler do not try to
fault neighbor pages ahead. So do not propagate

Neighbor (fault ahead) pages are never mapped with the wired attribute.

Wired faults are always "narrow". That means the fault handler do not try to
fault neighbor pages ahead. So do not propagate the `flt->wired' attribute to
the corresponding pmap_enter(9) calls and instead assert that it is false
whenever neighbor pages are entered in a memory space.

ok tb@

show more ...

Make uvmfault_anonget() return errno values instead of converting them.

ok miod@, tb@

Remove unused `fault_type' argument.

Use a helper to get lower page from backing store: uvm_fault_lower_io().

Reduce differences with NetBSD, no behavior change.

ok tb@, miod@

Do not put wired pages on the page queues & release their swap resources.

While here move the code to release swap resources outside of the pageq
mutex and shuffle some locking dances to reduce diff

Do not put wired pages on the page queues & release their swap resources.

While here move the code to release swap resources outside of the pageq
mutex and shuffle some locking dances to reduce differences with NetBSD.

ok miod@

show more ...

Check if the mapping for an vm_map_entry exists while holding its lock.

Prevent a race where the mapped object is being truncated while we are spinning
to unwire it.

Reported-by: syzbot+189cd03d088

Check if the mapping for an vm_map_entry exists while holding its lock.

Prevent a race where the mapped object is being truncated while we are spinning
to unwire it.

Reported-by: syzbot+189cd03d088cddbee591@syzkaller.appspotmail.com

Adapted from NetBSD r1.207, ok miod@

show more ...

Handle faults on wired map entries similarly to VM_FAULT_WIRE faults.

It is valid to fault on wired mappings if the object was truncated then grown
again.

Adapted from NetBSD r1.207, ok miod@

Revert previous, at least on arm64 too many pages end up being wired.

Do not put wired pages on the page queues & release their swap resources.

While here move the code to release swap resources outside of the pageq
mutex and shuffle some locking dances to reduce diff

Do not put wired pages on the page queues & release their swap resources.

While here move the code to release swap resources outside of the pageq
mutex and shuffle some locking dances to reduce differences with NetBSD.
ok miod@

show more ...