HAMER VFS - Cache additional b-tree nodes in directory.

* Cache inode record cursors in the directory at dip->cache[2] to reduce
cpu overhead when iterating a directory.

* Cache file object recor

HAMER VFS - Cache additional b-tree nodes in directory.

* Cache inode record cursors in the directory at dip->cache[2] to reduce
cpu overhead when iterating a directory.

* Cache file object record cursors in the file's governing directory at
dip->cache[3] after reading the last data record in a file, to reduce
cpu overhead when chain-reading files.

* Also add vfs.hammer.stats_btree_root_iterations, a counter which
is incremented whenever a B-Tree search is initiated from the root
node.

show more ...

HAMMER: adjustments to flush on umount

Instead of waiting for up to 25 hammer sync sequences before giving up,
instead wait up to 25 seconds before giving up.

Reported-by: Thomas Nikolajsen <thomas

HAMMER: adjustments to flush on umount

Instead of waiting for up to 25 hammer sync sequences before giving up,
instead wait up to 25 seconds before giving up.

Reported-by: Thomas Nikolajsen <thomas.nikolajsen@mail.dk>

show more ...

HAMMER performance and kernel memory issues.

* The flusher could stall-out due to deadlocks under certain heavy load
situations (blogbench -i1000 -o), causing the number of hammer inodes
to incr

HAMMER performance and kernel memory issues.

* The flusher could stall-out due to deadlocks under certain heavy load
situations (blogbench -i1000 -o), causing the number of hammer inodes
to increase slowly until the kernel runs out of memory.

Fix this by detecting the condition and stalling the threads
causing the deadlocks instead for a short while. Any user thread
which blocked performing B-Tree node I/O is a candidate.

* Improve the stability of the inode reclaim heuristic which slows
down the creation of new inodes when too many disconnected hammer
inodes are present.

* Remove vfs.hammer.limit_iqueued - this sysctl is no longer used.

Reported-by: YONETANI Tomokazu <qhwt+dfly@les.ath.cx>

show more ...

Use per-mount kmalloc pools for bulk data structures, particularly inodes
and records.

HAMMER Filesystem changes:

* Add version checking to the hammer mount code.

* Add ioctls to fetch the hammer filesystem version and to upgrade it,
to support future filesystem revisions.

* Rewri

HAMMER Filesystem changes:

* Add version checking to the hammer mount code.

* Add ioctls to fetch the hammer filesystem version and to upgrade it,
to support future filesystem revisions.

* Rewrite the algorithm used to integrate reservations with the blockmap
allocate/free code to fix a bug. hammer_buffers were sometimes not
being properly destroyed causing conflicts with later reallocations.

* Add support for up to four directory hash algorithms. Add flexibility
to the namekey iterator API to allow the algorithm to control the
iteration space.

* When iterating memory with the CURSOR_DELETE_VISIBILITY flag set, ignore
the record deleted flags.

HAMMER Utilities changes:

* Add two new hammer directives, 'version' and 'version-upgrade'.

* Change hammer namekey directive to 'namekey1' and 'namekey2', to support
generating directory hashes for different hashing algorithms.

* Change hammer cleanup to still prune for existing snapshots even if
new snapshot generation is disabled.

* Change hammer cleanup's default snapshots config for temporary
filesystems from 'snapshots 0d 60d' to 'snapshots 0d 0d'.

Reported-by: "Justin C. Sherrill" <justin@shiningsilence.com> (blockmap bug)

show more ...

HAMMER:

* Add version management ioctls to allow continued work on the filesystem
without interfering with production systems.

* Refuse to mount an unsupported version.

Raise the kmalloc limit for the M_HAMMER_INO pool based on desiredvnodes
to try to avoid kernel memory exhaustion panics.

Reported-by: Matthias Schmidt <matthias@dragonflybsd.org>, and others.

Change the autoflush code to autoflush when a flush group reaches a
certain point rather then trying to do it from the reclaim code. This
smooths out the flush sequences. The default is 2000, adjus

Change the autoflush code to autoflush when a flush group reaches a
certain point rather then trying to do it from the reclaim code. This
smooths out the flush sequences. The default is 2000, adjustable via
the vfs.hammer.autoflush sysctl.

Move hammer_inode_waitreclaims() calls. In particular, remove the calls
in the VOP_CLOSE() path. The problem with waiting for excessive reclaims to
drop in these paths is that the inode/vnode operation in question is probably
cached. Thus unrelated programs, even those just opening /dev/null (if /dev
is on a HAMMER filesystem), can wind up blocking in hmrrcm for no good reason.

Instead defer the hammer_inode_waitreclaims() call to the end of the
transaction code if a new inode had to be created during the transaction.
Thus we tend to block on operations that did not have previously cached
vnodes to work with instead of operations on cached vnodes.

Reported-by: Hasso Tepper <hasso@estpak.ee>

show more ...

* Implement the ability to export NULLFS mounts via NFS.

* Enforce PFS isolation when exporting a HAMMER PFS via a NULLFS mount.

NOTE: Exporting anything other then HAMMER PFS root's via nullfs doe

* Implement the ability to export NULLFS mounts via NFS.

* Enforce PFS isolation when exporting a HAMMER PFS via a NULLFS mount.

NOTE: Exporting anything other then HAMMER PFS root's via nullfs does
NOT protect the parent of the exported directory from being accessed via NFS.

Generally speaking this feature is implemented by giving each nullfs mount
a synthesized fsid based on what is being mounted and implementing the
NFS export infrastructure in the nullfs code instead of just bypassing those
functions to the underyling VFS.

show more ...

HAMMER: Mirroring, misc bug fixes

* Adjust hammer_flusher_async() to queue an extra flush if called twice
in quick succession. This fixes the 'sync' command to properly sync
the entire filesyte

HAMMER: Mirroring, misc bug fixes

* Adjust hammer_flusher_async() to queue an extra flush if called twice
in quick succession. This fixes the 'sync' command to properly sync
the entire filesytem. Previously two syncs were needed.

* Fix a bug where a user application could get stuck due to HAMMER
losing track of an inode flush.

* Mirroring masters now use the most recent fully committed transaction
id instead of the last flushed (but still subject to rollback) tid.

This fixes an issue where a mirror could pass information still subject
to crash recovery rollback to the slave. Now only fully committed
information is passed to the slave.

* Fix a transitory bug where the mirroring code would sometimes not
sync the correct delete state to the slave. The slave would always
be corrected in the next pass, however. Now the slave is correct
at all times.

* Fix a bug in hammer_mirror_write() where a delete-to operation could
livelock.

* Add a new HAMMER ioctl which waits for the committed data transaction
id to change. This will be used by the mirroring code to implement
continuous streaming operation.

Reported-by: Francois Tigeot <ftigeot@wolfpond.org>,
Michael Neumann <mneumann@crater.dragonflybsd.org>
(the user application freeze bug)

show more ...

HAMMER - fix kmalloc exhaustion w/ 3G ram

* Give the hammer_inode structure its own kmalloc pool.

Implement mounting of a Hammer filesystem as a root filesystem.

To mount a Hammer filesystem as root filesystem (e.g. /dev/ad0s1d),
you have to specify the following in /boot/loader.conf:

vfs.r

Implement mounting of a Hammer filesystem as a root filesystem.

To mount a Hammer filesystem as root filesystem (e.g. /dev/ad0s1d),
you have to specify the following in /boot/loader.conf:

vfs.root.mountfrom="hammer:ad0s1d"

show more ...

HAMMER commit

* Discard recovered buffers when unmounting a read-only HAMMER mount.

* Discard recovered buffers when an attempted HAMMER mount fails.

* Clean up a transaction that was left open wh

HAMMER commit

* Discard recovered buffers when unmounting a read-only HAMMER mount.

* Discard recovered buffers when an attempted HAMMER mount fails.

* Clean up a transaction that was left open when a HAMMER mount fails to
locate the root inode.

* Check that an undo record is at least minimally sized so corruption
does not result in an infinite loop.

Reported-by: Dionysus Blazakis <dion.blazakis@gmail.com>,
Gergo Szakal <bastyaelvtars@gmail.com>

show more ...

HAMMER 65/Many: PFS cleanups and confusion removal

* Simplify master/slave operation. Remove the master=, slave, and no-mirror
options from pfs-update. Retain pfs-master, pfs-slave, pfs-upgrade,

HAMMER 65/Many: PFS cleanups and confusion removal

* Simplify master/slave operation. Remove the master=, slave, and no-mirror
options from pfs-update. Retain pfs-master, pfs-slave, pfs-upgrade, and
pfs-downgrade for dealing with master and slave operation.

Move the master= and nomirror specifications back to hammer_mount. They
have to be mount-wide anyhow, the current code does not support different
master ID's on a PFS-by-PFS basis.

* Add some additional failsafes to the mount code. The master id cannot
be allowed to change by a mount update (-u).

* Fix a bug in hammer pfs-upgrade operation that could result in a livelock
during the B-Tree iteration.

show more ...

HAMMER 63/Many: IO Error handling features

This commit removes all the remaining Debugger() calls and KKASSERTs in the
I/O error path. Errors are now propagated up the call tree and properly
report

HAMMER 63/Many: IO Error handling features

This commit removes all the remaining Debugger() calls and KKASSERTs in the
I/O error path. Errors are now propagated up the call tree and properly
reported.

* Report I/O errors instead of asserting.

* Read or Write errors in the flush path disable flushing and force the mount
into read-only mode. Modified buffers are left locked in memory until
umount to provide a consistent snapshot of the state of the filesystem.

You must umount and remount to recover the filesystem. The filesystem
will automatically rollback to the last valid flush upon remounting.

* umount and umount -f are now able to unmount a HAMMER filesystem that has
catastrophic write errors (e.g. pulling the USB cable on an external
drive).

show more ...

HAMMER 61E/Many: Features

* Implement hammer iostats.

HAMMER 61F/Many: Stabilization w/ simultanious pruning and reblocking

* BUG FIX: When doing direct-read check to see if any device buffers
are aliasing the disk block and flush any we find which a

HAMMER 61F/Many: Stabilization w/ simultanious pruning and reblocking

* BUG FIX: When doing direct-read check to see if any device buffers
are aliasing the disk block and flush any we find which are dirty.
This ensures that reblocked data gets to disk before a direct-read
tries to read them FROM the disk.

* BUG FIX: Fix a bug introduced in a recent commit where the flusher
wlll not always completely flush the UNDO FIFO or completely flush
all meta-data, resulting in a rollback after a normal umount/mount.

* BUG FIX: Direct-writes queue I/O independant of the in-memory record.
When the backend flusher flushes the record, making it available in the
B-Tree, make sure that the indepent I/O has completed. Otherwise
a later reblocking operation might read the media before the direct-write
has actually completed.

* BUG FIX: In-memory records are not subject direct-IO, since their data
is not yet on the media.

* BUG FIX: Do not allow mount to succeed unless all volumes have been found.
(Reported-by: Sascha Wildner <saw@online.de>)

* BUG FIX: The bd_heatup() call in the reblocker was in the wrong place,
potentially causing the cursor to shift unexpectedly.

* Reorient some of the buffer invalidation code by enhancing
the reservation code.

* Add read CRC verification logic for some direct-reads, but comment it out
because the VM system's bogus-page replacement breaks it.

show more ...

HAMMER 61E/Many: Stabilization, Performance

* The UNDO FIFO could get over-full, causing an assertion. This was because
certain load tests could cuase directories to end up with tens of
thousan

HAMMER 61E/Many: Stabilization, Performance

* The UNDO FIFO could get over-full, causing an assertion. This was because
certain load tests could cuase directories to end up with tens of
thousands of directory entries and all of them were being flushed in
a single flush group.

Rewrite the flush group infrastructure to fix the issue. Instead of
having a two-stage flush we now have flush compartmentalization with
the new hammer_flush_group structure. Flushes can thus be broken down
into transactions that don't blow out the UNDO FIFO.

* Improve flush performance by quite a bit. The new infrastructure
generates fewer B-Tree conflicts and appears to flush the work more
smoothly, probably owing to the fact that the upper size of each
flush group is now bounded.

show more ...

HAMMER 61A/Many: Stabilization

* Fix a buffer exhaustion issue. When creating large numbers of empty files
a single inode sync of the directory can exhaust the buffer cache because
not enough o

HAMMER 61A/Many: Stabilization

* Fix a buffer exhaustion issue. When creating large numbers of empty files
a single inode sync of the directory can exhaust the buffer cache because
not enough other things get dirty enough to force a flush.

Put a check in the record syncing code and do a partial finalization if
necessary to avoid deadlocking the buffer cache.

* Fix a panic caused by a missing call to hammer_cursor_deleted_element().

Reported-by: Michael Neumann <mneumann@ntecs.de>,
Gergo Szakal <bastyaelvtars@gmail.com>

show more ...

HAMMER 60E/Many: Mirroring, bug fixes

* Work on the mirror_tid propagation code. The code now retries on
EDEADLK so propagation is guaranteed to reach the root.

* Get most of the mirror_write co

HAMMER 60E/Many: Mirroring, bug fixes

* Work on the mirror_tid propagation code. The code now retries on
EDEADLK so propagation is guaranteed to reach the root.

* Get most of the mirror_write code working.

* Add PFS support for NFS exports. Change fid_reserved to fid_ext and use
it to store the localization parameter that selects the PFS. This isn't
well tested yet.

* BUGFIX: Fix a bug in vol0_last_tid updates. Flush sequences might
not always update the field, creating issues with mirroring and snapshots.

* BUGFIX: Properly update the volume header CRC.

* CLEANUP: Fix some obj_id's that were u_int64_t's. They should be int64_t's.

* CLEANUP: #if 0 out unused code, remove other bits of unused code.

show more ...

HAMMER 60B/many: Stabilization

* Redo the estimated space usage calculation, it was horribly broken.
There may still be issues with read+write memory maps.

* Fix a bug in the reblocker that allow

HAMMER 60B/many: Stabilization

* Redo the estimated space usage calculation, it was horribly broken.
There may still be issues with read+write memory maps.

* Fix a bug in the reblocker that allowed it to exhaust all available
space on the media. While the reblocker eventually compacts the media
it can actually require additional disk space while running.

Note that file deletions in normal history retention mode do not recover
any space unless followed by pruning, and may even require additional space.
If worse comes to worse a full filesystem might have to be remounted in
nohistory mode to recover space.

Reported-by: Gergo Szakal <bastyaelvtars@gmail.com>

show more ...

HAMMER 60A/many: Mirroring work

* Add a pseudo-fs structure for tracking pseudo-fs configurations.

* Add pseudo-fs status and configuration ioctls

* Fix a b-tree node deletion race that could resu

HAMMER 60A/many: Mirroring work

* Add a pseudo-fs structure for tracking pseudo-fs configurations.

* Add pseudo-fs status and configuration ioctls

* Fix a b-tree node deletion race that could result in an
assertion (reported by Rumko <rumcic@gmail.com>)

show more ...

HAMMER 59F/Many: Stabilization pass

* Use kmalloc_limit() to calculate a better record limit. This fixes an
issue on machines with limited system memory where HAMMER could blow
out its kmalloc

HAMMER 59F/Many: Stabilization pass

* Use kmalloc_limit() to calculate a better record limit. This fixes an
issue on machines with limited system memory where HAMMER could blow
out its kmalloc pool by allocating too many records.

show more ...

HAMMER 59D/Many: Sync with buffer cache changes in HEAD.

* Adjust hammer to limit dirty meta-data buffers based on total bytes
rather then total buffers.

* Limit to 1/4 the buffer cache limit (fo

HAMMER 59D/Many: Sync with buffer cache changes in HEAD.

* Adjust hammer to limit dirty meta-data buffers based on total bytes
rather then total buffers.

* Limit to 1/4 the buffer cache limit (for now)... a better solution
is needed.

show more ...

HAMMER 59B/Many: Stabilization pass - fixes for large file issues

* The flusher was trying to flush whole inodes in one pass. If a large
(in the 100GB range) file is deleted or truncated this sin

HAMMER 59B/Many: Stabilization pass - fixes for large file issues

* The flusher was trying to flush whole inodes in one pass. If a large
(in the 100GB range) file is deleted or truncated this single pass could
blow out the UNDO FIFO and cause a panic.

Detect the case and only do a partial flush, then requeue the inode to
the next flush group and reflush. This continues until the file is
completely eradicated or the truncation has completed.

* In addition, the same flusher issue could exhaust the kernel's buffer
cache with unflushable dirty buffers. The same fix applies.

* The UNDO FIFO code was calculating available space based on the next
flush's start position instead of the last flush's start position,
allowing areas of the UNDO FIFO still subject to a recovery pass
(if a crash were to occur right then) to be overwritten too soon.

* The pruner and reblocker were doing insufficient UNDO FIFO space checks
and could blow out the UNDO FIFO. Fixed.

show more ...