libdmsg - Get the encryption operational again

* Currently encrypts/decrypts, but the algorithm is really just a
place-holder for something better. It does not use any openssl
algos beyond basi

libdmsg - Get the encryption operational again

* Currently encrypts/decrypts, but the algorithm is really just a
place-holder for something better. It does not use any openssl
algos beyond basic public key exchange, session key exchange, and
raw aes-256-gcm encryption with a block IV increment to prevent replay
attacks.

* Note that in the final protocol there will be two verifiers embedded
in the dmsg itself, rather than tacked on via the transport. One is
the 32-bit header crc (there is also an aux-data crc), and the second
is a 64-bit verifier that the link-level is intended to replace and check.
The dmsg also has a signature and 24 random bits to mix things up.

show more ...

lib/libdmsg: Fix compile warning on Linux distros

Appeared since 030e3428e0c23c47cef4dfab51f988a8fa665d19.

--
msg.c:202: warning: pointer targets in passing argument 1 of 'atomic_set_int' differ in

lib/libdmsg: Fix compile warning on Linux distros

Appeared since 030e3428e0c23c47cef4dfab51f988a8fa665d19.

--
msg.c:202: warning: pointer targets in passing argument 1 of 'atomic_set_int' differ in signedness
atomic.h:129: note: expected 'volatile u_int *' but argument is of type 'int *'

show more ...

dmsg - Formalize most of the debugging output, cleanup, fix uninit bug

* Macroize and formalize most of the debugging output.

* Add LNK_PING support.

* Fix a bug in kdmsg_iocom_uninit(). The RX t

dmsg - Formalize most of the debugging output, cleanup, fix uninit bug

* Macroize and formalize most of the debugging output.

* Add LNK_PING support.

* Fix a bug in kdmsg_iocom_uninit(). The RX thread can stall reading
from the pipe and never wakeup to see the KILLRX flag if no traffic
comes in.

To deal with this kdmsg_iocom_uninit() sends a LNK_PING over the link,
then gives the threads 10 seconds to shutdown nicely. If they do not
the pipe descriptor is not-so-nicely shutdown().

show more ...

dmsg - Stabilization work

* Refactor the circuit failure handling code. When a connection is lost
circuits running through that connection and all sub-circuits/states
(recursively) are aborted.

dmsg - Stabilization work

* Refactor the circuit failure handling code. When a connection is lost
circuits running through that connection and all sub-circuits/states
(recursively) are aborted. This will propagate through the graph and
there are plenty of edge cases where a failure may be propagating in
one direction and a request in the other direction.

The library is responsible for providing missing transaction closures
when circuits and states fail.

* Add code to handle circuit failure races against newly created messages.
The STATE_DYING flag is now inherited by the newly created message from
its parent.

* The state structure on receive is now updated before the callback is
made so it can be atomic with the lock, instead of after.

* Lots of debugging added and some cleanup.

show more ...

dmsg - Stabilization work

* Add a refs field to dmsg_state and kdmsg_state for retention and
disposal.

* Separate out the tracking of state->subq and the state RB trees.

* Greatly simplify the i

dmsg - Stabilization work

* Add a refs field to dmsg_state and kdmsg_state for retention and
disposal.

* Separate out the tracking of state->subq and the state RB trees.

* Greatly simplify the iocom shutdown code and functions related
to handling communications failures. When iterating states for
shutdown, which requires simulating a received failure message,
we can now simply iterate via state->subq.

* Greatly simplify how the simulated failures are generated and
handled (See dmsg_msg_simulate_failure()). This is probably the
most complex part of the library.

* Fix a memory leak in kern_dmsg.c.

* Replace xdisk's per-softc token with a lockmgr lock. Atomicy
has to be guaranteed across blocking conditions in certain cases
and it was easier to simply use a lock for everything.

Cleanup the locking.

* Ripout the shutdown check in the I/O path, which can deadlock
the disk management threads. This will need to be revisited
as it means that /dev/xa* and /dev/serno* devices remain in
/dev after a link failure even when there are no opens on the
device.

* Add the B_FAILONDIS flag to struct buf. This flag allows the disk
probe code to tell xdisk that it is ok for the I/O to fail, allowing
xdisk to discard I/Os that would otherwise block or deadlock the
disk probe code when the related network connection is lost.

show more ...

libdmsg - Fix incomplete write bug

* Fix a bug where a partial non-blocking write of encrypted data does not
properly set the WREQ flag if the related dmsg has been disposed of.
This could cause

libdmsg - Fix incomplete write bug

* Fix a bug where a partial non-blocking write of encrypted data does not
properly set the WREQ flag if the related dmsg has been disposed of.
This could cause random stalls of the link.

* Fix an errno-handling bug.

* Cleanup and debugging.

show more ...

libdmsg - Fix memory leak

* Fix a memory leak in the message relay code. When relaying a message
we just transfer the aux_data pointer to the relay message. The
allocation of the relay message

libdmsg - Fix memory leak

* Fix a memory leak in the message relay code. When relaying a message
we just transfer the aux_data pointer to the relay message. The
allocation of the relay message was also allocate aux_data space which
was getting thrown away and not freed.

show more ...

libdmsg - Fix buffer indexing bug in crypted path

* Fix a buffering index bug in the crypted path which causes a buffer
overrun and/or implodes the connection on a protocol error.

libdmsg - Fix buffer overrun

* Large messages (generally large data payloads) could overrun the FIFO
buffer. e.g. dd if=/dev/xa<blah> of=/dev/null bs=32k

* Fix by chunking up large payloads.

hammer2 - dmsg stabilization

This is a general stabilization pass testing xdisk resumption on various
communications failures (killing the local or remote hammer2 service demon).

* Simulate failure

hammer2 - dmsg stabilization

This is a general stabilization pass testing xdisk resumption on various
communications failures (killing the local or remote hammer2 service demon).

* Simulate failures on recursive transactions in a bottom-up fashion

* bcmp() does not return an inequality (neg, zero, or positive). It
returns only 0 or 1. Change fs_label RBTREE test to strcmp()

* ref-count the state structure, use dmsg_state_hold()/dmsg_state_drop()
to control it. Fix at least one memory leak.

* Consolidate simulated transaction failure core into a helper
function dmsg_msg_simulate_failure().

* Clean up the tx state on the front-end after the msg has extracted
necessary information from it, instead of in the transmit thread.
This is so the state fields are updated synchronously and not delayed
by the transmit queue.

* Change xdisk's error test to avoid accidently failing a BIO due to
a communications issue (verses an actual IO error on the remote physical
disk).

* Remove some invalid assertions, add assertions.

show more ...

hammer2 - Retool dmsg mechanics to improve virtual circuit design 2/2

* Use transaction stacking to effectively connect resources to consumers.
Issuing a transaction over a received SPAN effective

hammer2 - Retool dmsg mechanics to improve virtual circuit design 2/2

* Use transaction stacking to effectively connect resources to consumers.
Issuing a transaction over a received SPAN effectively creates a virtual
circuit to the originator.

* The hammer2 service demon (userland) is currently being used to route,
but direct kernel<->kernel socket connections are now theoretically
possible for later performance work. Current performance is going
to be low due to layering.

* Get subr_diskiocom and the xdisk driver mostly working again. The
code is a lot cleaner but still needs considerable stability work and
better unwinding during failure conditions.

* Start work on hammer2<->hammer2 communications.

show more ...

hammer2 - Retool dmsg mechanics to improve virtual circuit design 1/2

* Rip-out the circuit structures and forging code. These changes simplify
the DMSG code considerably.

* Retool the core comm

hammer2 - Retool dmsg mechanics to improve virtual circuit design 1/2

* Rip-out the circuit structures and forging code. These changes simplify
the DMSG code considerably.

* Retool the core command/response messaging mechanics to allow either
side of a transaction to initiate commands and receive responses.

This means we cannot use DMSGF_REPLY to determine whether the transmit-side
or receive-side state RBTREE holds the msgid. Instead we add two more
flags DMSGF_REVTRANS and DMSGF_REVCIRC to tell the receiver which RBTREE
holds the msgid and/or circuit id.

* Retool to allow transaction stacking. Sub-transactions can now run under
their parents.

* Retool the transaction code to provide virtual circuit functionality
through the use of transaction stacking.

With these changes, the normal SPAN mechanism which operates using
open transactions can also be used to route messages over the SPAN.
There is no longer a need to forge a return path because sub-transaction
commands can now be initiated 'out' over an active, received SPAN
transaction.

This part is not completely working yet, it needs the actual routing
code and some adjustments to the SPAN mechanism to prevent path ripups
from interfering with any in-progress transactions. Ultimately the
availability of a new path would have two be detected by the end points
so new 'connections' can be forged over the new, better path.

show more ...

hammer2 - more dmsg/separation work

* More separation work in libdmsg, kern_dmsg.c, and kernel dmsg consumers.

* subr_diskiocom.c should no longer need hammer2_disk.h (sys/dmsg.h)
for the kernel

hammer2 - more dmsg/separation work

* More separation work in libdmsg, kern_dmsg.c, and kernel dmsg consumers.

* subr_diskiocom.c should no longer need hammer2_disk.h (sys/dmsg.h)
for the kernel build.

* Add dmsg->tcmd to smooth some switch() use cases.

show more ...

libdmsg - refactor remove hammer2-specific code

* Remove hammer2-specific code.

* Refactor the callback API to allow the hammer service code to handle
the H2-specific LNK_CONN commands... mostly

libdmsg - refactor remove hammer2-specific code

* Remove hammer2-specific code.

* Refactor the callback API to allow the hammer service code to handle
the H2-specific LNK_CONN commands... mostly VOLCONF.

show more ...

merge

cluster - Stabilization

* Fix disconnect/reconnect sequence for autoconn (/etc/hammer2/autoconn).
The pipe used to signal termination of the iocom_core() was not supposed
to be closed by iocom_d

cluster - Stabilization

* Fix disconnect/reconnect sequence for autoconn (/etc/hammer2/autoconn).
The pipe used to signal termination of the iocom_core() was not supposed
to be closed by iocom_done().

* The shutdown code now simply sets DMSG_IOCOMF_EOF instead of trying to
shutdown() the socket.

* Fix double mutex lock in dmsg_msg_alloc().

show more ...

cluster - misc cleanup

* Rename the routing function to something that I remember.

cluster - Improve hammer2 connect directive, add /etc/hammer2/autoconn

* Improve the hammer2 connect directive, fixing a socket descriptor reuse
race.

* Monitor the file /etc/hammer2/autoconn whi

cluster - Improve hammer2 connect directive, add /etc/hammer2/autoconn

* Improve the hammer2 connect directive, fixing a socket descriptor reuse
race.

* Monitor the file /etc/hammer2/autoconn which contains a list of hosts
the hammer2 service demon whould automatically maintain connections to.

* Adjust the dmsg_master_service() API to also allow an alternative
signalling descriptor and callback to be specified. This simplifies
forced connection terminations.

* Fix descriptor leak in dmsg_master_service() on closure.

show more ...

cluster - more libdmsg work

* Fix numerous bugs, including a nasty edge case in the encryption code.

* Do a better job locking things (state structures still need help).

cluster - Major kernel component work (diskiocom, xdisk, kdmsg)

* Major implementation and stabilization work. Fix lots of bugs,
improve the AUTO flags and APIs.

* xdisk and diskiocom are now op

cluster - Major kernel component work (diskiocom, xdisk, kdmsg)

* Major implementation and stabilization work. Fix lots of bugs,
improve the AUTO flags and APIs.

* xdisk and diskiocom are now operational for the first time.

xdisk is a consumer of remote block devices. You pass it one end of
a pipe and use an ioctl to create /dev/xa* devices. The kernel's
nominal disk management subsystem handles /dev/serno/* and will also
probe the /dev/xa* devices to create the slices.

diskiocom is a block device service. An ioctl is used to connect a
raw disk device to the network via a pipe.

* The hammer2 service demon ties the two together over the network.

* Initial testing with a local hammer2 service daemon looping da0 to xa0
succeeded, ~46MB/sec unoptimized using dd.

show more ...

cluster - circuit tracking, bug fixes

* Misc virtual circuit tracking work and bug fixes

hammer2 - cluster / libdmsg circuit work

* Major work on the virtual circuit code. Note there are still some thread
races.

* Major work on the spanning tree code to allow for future symmetric

hammer2 - cluster / libdmsg circuit work

* Major work on the virtual circuit code. Note there are still some thread
races.

* Major work on the spanning tree code to allow for future symmetric
pathing support.

* fs_label is now part of the RB tree compare, along with pfs_fsid.
(e.g. the serial number for BLK protocol services).

* Rip out the 'router' structure, replace with the 'circuit' structure.
Embed circuit0 in the iocom.

show more ...

hammer2 - dmsg blockdev work (cleanup)

* Cleanup

hammer2 - Messaging layer separation work part 5

* Fix debug shell callback issue

hammer2 - Messaging layer separation work part 2

* Move most of the low-level message handling from the hammer2 utility
to a new library called libdmsg.

* Move additional message headers from the

hammer2 - Messaging layer separation work part 2

* Move most of the low-level message handling from the hammer2 utility
to a new library called libdmsg.

* Move additional message headers from the hammer2 vfs code to libdmsg,
cleanup.

* Rename and reorganize a ton of stuff.

show more ...