kernel/hammer: Remove the old inactive live dedup code.

It had been disabled in 6de803f0514ff0019181d026a825d1e8a9a4c404
in 2012 due to issues with it.

sys/vfs/hammer: Use typedef'd for struct hammer_mount*

The whole hammer code is mix of using struct and typedef'd.
Use typedef'd because majority of the code use typedef'd.

The last 17 commits incl

sys/vfs/hammer: Use typedef'd for struct hammer_mount*

The whole hammer code is mix of using struct and typedef'd.
Use typedef'd because majority of the code use typedef'd.

The last 17 commits including this one don't affect binary.

show more ...

sys/vfs/hammer: Remove validate_zone()

Code becomes less clear with this function and usage of sum of results.
Just check if the given offsets are data zones or not (unless we plan
to support non-da

sys/vfs/hammer: Remove validate_zone()

Code becomes less clear with this function and usage of sum of results.
Just check if the given offsets are data zones or not (unless we plan
to support non-data zones, but we don't).

show more ...

sys/vfs/hammer: Add HAMMER_ZONE()

sys/vfs/hammer: Add hammer_btree_extract_data() [1/2]

hammer_btree_extract() reads data from block devices if flag
has HAMMER_CURSOR_GET_DATA. It doesn't really matter if the flag
is HAMMER_CURSOR_G

sys/vfs/hammer: Add hammer_btree_extract_data() [1/2]

hammer_btree_extract() reads data from block devices if flag
has HAMMER_CURSOR_GET_DATA. It doesn't really matter if the flag
is HAMMER_CURSOR_GET_DATA or (LEAF | DATA) as long as the flag
has HAMMER_CURSOR_GET_DATA. Calling this function will cause
cursor->leaf to point to the node element in question regardless
of the flag value.

This commit adds hammer_btree_extract_data() which is just a
wrapper over hammer_btree_extract(cursor, LEAF | DATA).
This commit replaces those with (LEAF | DATA).

show more ...

sys/vfs/hammer: Adjust raw kprintfs using hkprintf variants

This is part3 of hkprintf related after the following two.
sys/vfs/hammer: Change hkprintf() to macro and add variants [2/2]
sys/vfs/hamme

sys/vfs/hammer: Adjust raw kprintfs using hkprintf variants

This is part3 of hkprintf related after the following two.
sys/vfs/hammer: Change hkprintf() to macro and add variants [2/2]
sys/vfs/hammer: Change hkprintf() to macro and add variants [1/2]

Above two commits have replaced the existing kprintf calls
using "HAMMER:" or "HAMMER(label)" or function name prefix
with hkprintf and newly added variants, which basically didn't
change actual output other than fixing wrong function names
to the right ones, etc.

This commit continues replacing remaining kprintfs to make
output more understandable than raw kprintf calls with no clue
that they're hammer related.

For example, an error message like
"BIGBLOCK UNDERFLOW\n"
or a debug message like
"rt %3u, xt %3u, tt %3u\n"
become more understanbale with "HAMMER:" prefix or the name
of the function.

This commit is based on the followings.
1. Use hdkprintf which is hkprintf variant with __func__ prefix
if that kprintf call is used when vfs.hammer.debug_xxx is
enabled. This implies the messages are only for debugging
and those are usually better and more understandable with
a function name prefix as mentioned above. Also this is
what's been done mostly in the existing hammer code.
2. Use hkprintf which has "HAMMER:" prefix if that kprintf
call is a regular hammer message that appears in regular
filesystem operations such as
"Formatting of valid HAMMER volume %s denied. Erase with dd!\n".
3. Use h[vm]kprintf which are hkprintf variants with hammer
label prefix "HAMMER(label)" if that kprintf can safely
access the label via vol or hmp pointer. Some kprintfs in
hammer does this rather than just "HAMMER:" and this seems
to be better, however this commit doesn't go far as to
aggressively replace the existing ones with this because
a caller safely dereferencing hmp or vol is a different
topic from merely replacing.

show more ...

sys/vfs/hammer: Change hkprintf() to macro and add variants [1/2]

This commit does the following.

1. Remove an inline function hkprintf().
2. Add a macro version of hkprintf() and its variants that

sys/vfs/hammer: Change hkprintf() to macro and add variants [1/2]

This commit does the following.

1. Remove an inline function hkprintf().
2. Add a macro version of hkprintf() and its variants that
embed the following prefix.
"HAMMER: ..."
"HAMMER(label) ..."
3. Replace raw kprintf() calls that have above prefix with
newly added macros.

hkprintf() macro doesn't use hammer_debug_debug flag that was
used by the inline function version of hkprintf(), but this
doesn't make much difference from the way hkprintf() and
hammer_debug_debug have been used.

show more ...

sys/vfs/hammer: Use either HAMMER: or hammer:

Should be either of the following.
kprintf("HAMMER: ...\n");
kprintf("hammer: ...\n");

The existing code mostly uses "HAMMER:" so use "HAMMER:".
Assumi

sys/vfs/hammer: Use either HAMMER: or hammer:

Should be either of the following.
kprintf("HAMMER: ...\n");
kprintf("hammer: ...\n");

The existing code mostly uses "HAMMER:" so use "HAMMER:".
Assuming no one or no userspace triggers anything via
"hammer:" string in dmesg.

show more ...

kernel/hammer: Simplify an if/goto/else construct.

hammer: Cleanups

Unfold lines (that aren't intentionally separated
into different lines) that fit in 80 chars.

Fold lines that are way too long.

sys/vfs/hammer: Change "big block" to "big-block"

- This word refers to 8MB chunk in hammer's blockmap layers,
not literally "big" "block".

- Changes are mostly comments and some in printf and ha

sys/vfs/hammer: Change "big block" to "big-block"

- This word refers to 8MB chunk in hammer's blockmap layers,
not literally "big" "block".

- Changes are mostly comments and some in printf and hammer(8).

- The official design document as well as much of the existing
code (excluding variable and macro names) use "big-block".
https://www.dragonflybsd.org/hammer/hammer.pdf

- Also see e04ee2de.

show more ...

hammer: fix terminology of "large block"

This cleanup patch changes terminology "large block" to "big block".

- Both "large block" and "big block" are widely used in hammer source from
kernel to us

hammer: fix terminology of "large block"

This cleanup patch changes terminology "large block" to "big block".

- Both "large block" and "big block" are widely used in hammer source from
kernel to userspace, however these two refer to the same data structure which
is a 8MB sized chunk within low level blockmapped storage layer.

- The original design document https://www.dragonflybsd.org/hammer/hammer.pdf
uses big block for this data structure. Having two expressions in its
implementation is confusing and makes grep difficult.

Closes: #2782

show more ...

kernel - Fix 3:00 a.m. crashes (deadlocks) related to HAMMER VM use

When memory is low and the pageout daemon needs to write things out we still
need to have at least some reserve to perform the sup

kernel - Fix 3:00 a.m. crashes (deadlocks) related to HAMMER VM use

When memory is low and the pageout daemon needs to write things out we still
need to have at least some reserve to perform the supporting operations for
the pageout. HAMMER is particularly memory intensive and could get into a
situation where insufficient reserve memory was available, deadlocking the
system.

With these changes DragonFly should run stable on systems with as little
as 256M of ram, and possibly a bit lower.

* The getblk/bread/bwrite/etc brelse/bqrelse sequence used to manage buffers
had several bugs in it that prevented the low memory handling code from
operating properly. The b[q]relse() sequence was not properly detecting
the low memory condition and freeing or caching the underlying VM pages
(when possible).

* Also change the low memory test used by the buffer cache from 'severe'
to 'min' in kern/vfs_bio.c. We may be able to change this back to 'severe'
at a later date with further testing. These tests are in brelse(),
bqrelse(), and vfs_vmio_release().

* Rewrite bio_page_alloc(). It effectively does the same thing that it did
before but should operate more smoothly. We also no longer try to recover
pages from unrelated buffer cache buffers from this function, which could
lead to deadlocks. The warning kprintf is now also rate-limited.

* Add a buffer overload test in the hammer dedup ioctl. A hammer dedup
could cause a buffer cache deadlock by allowing too many dirty buffers
to build up.

* Add a VM memory test to the core hammer flusher code that was previously
only checking for the UNDO meta-data and buffer overload limits. This
is now done on a per-record basis and should prevent HAMMER from allocating
too much memory during a flusher operation when the VM system is already
too low on memory.

* Add some vm_wait_nominal() calls in critical I/O paths, but make sure we
do not use these calls in any I/O path used by the HAMMER pageout code.

Probably the most important path is the vm_object_page_clean*() code
path, effectively called via either msync() or via the 30-60 second
system sync.

* Properly bawrite() a buffer in hammer_vop_write() when IO_ASYNC is set
(which is used by the pageout daemon), otherwise the pageout daemon will
not be able to directly recover memory in low memory situations when
paging to a HAMMER file mapped SHARED/RW.

Testing-by: tuxillo, lentferj, ftigeot, dillon

show more ...

HAMMER VFS - Force reading from the block device when live dedup is enabled

* Attempting to access the vnode's buffer cache is currently not MPSAFE
and can cause crashes so don't try to do that.

HAMMER VFS - Force reading from the block device when live dedup is enabled

* Attempting to access the vnode's buffer cache is currently not MPSAFE
and can cause crashes so don't try to do that. Use the block device
instead.

* It is suggested but not required that vfs.hammer.double_buffer mode also
be enabled if you use vfs.hammer.live.deup.

show more ...

HAMMER VFS - vfs.hammer.double_buffer adjustments

* Do not try to use the (currently broken) _vnode_validate() to
validate de-dup data when double_buffer is turned on. Use the
device buffer cac

HAMMER VFS - vfs.hammer.double_buffer adjustments

* Do not try to use the (currently broken) _vnode_validate() to
validate de-dup data when double_buffer is turned on. Use the
device buffer cache instead.

show more ...

HAMMER - Add live_dedup_cache_size sysctl

* There is now a hammer_live_dedup_cache_size variable which can be
set via sysctl vfs.hammer.live_dedup_cache_size. The default is 4096.

* A better sol

HAMMER - Add live_dedup_cache_size sysctl

* There is now a hammer_live_dedup_cache_size variable which can be
set via sysctl vfs.hammer.live_dedup_cache_size. The default is 4096.

* A better solution is to make it scale automatically according to
nbufs or a number of vnodes.

show more ...

HAMMER - Add live dedup sysctl and support

* Adds *experimental* live dedup (aka efficient cp) support

* sysctl vfs.hammer.live_dedup
0 - disabled (default)
1 - enabled, populate dedup cach

HAMMER - Add live dedup sysctl and support

* Adds *experimental* live dedup (aka efficient cp) support

* sysctl vfs.hammer.live_dedup
0 - disabled (default)
1 - enabled, populate dedup cache on reads
2 - enabled, populate dedup cache on reads and writes

show more ...

HAMMER - Add hammer dedup filesystem version dump

* Require a version 5 filesystem for dedup operation. Upgrading can be done
in-place. All this does is make the layer2 bytes_free field a signed

HAMMER - Add hammer dedup filesystem version dump

* Require a version 5 filesystem for dedup operation. Upgrading can be done
in-place. All this does is make the layer2 bytes_free field a signed
value instead of unsigned, allowing it to go negative.

This allows dedup to use the field to account for multiple references to
the same block of data, causing the field to (likely) go negative.

show more ...

HAMMER - Add hammer dedup directive and support

* Implements all the logic required to dedup a HAMMER filesystem.

* There is one remaining issue and that is the reblocker's propensity to
undo de-

HAMMER - Add hammer dedup directive and support

* Implements all the logic required to dedup a HAMMER filesystem.

* There is one remaining issue and that is the reblocker's propensity to
undo de-dup's hard work in certain cases.

* Code bounty for hammer_dedup

Submitted-by: Ilya Dryomov <idryomov@gmail.com>

show more ...