ip_forward: Migrate cpu if hash doesn't match.

Packet filter re-writes can cause the call to ip_forward to be on the
wrong CPU. Detect this case and correct it. Check M_HASH at the
beginning of ip_i

ip_forward: Migrate cpu if hash doesn't match.

Packet filter re-writes can cause the call to ip_forward to be on the
wrong CPU. Detect this case and correct it. Check M_HASH at the
beginning of ip_input and dispatch to a new CPU if we aren't in the
right place. This mirrors what is done for packets that are destined to
the transport layer. This causes ip_forward and ip_output to be called
on the correct CPU, including any states that are created by output
rules.

show more ...

kernel - Make getpeereid() work with socketpair()

* getpeereid() now works with socketpair() sockets.

Requested-by: rtzoeller

mbuf: Add message header for generic mbuf sending/receiving.

tcp: Make setsockopt(2) TCP_{NODELAY,NOPUSH,NOOPT,FASTKEEP} asynchronous

This makes all network system calls used by nginx asynchronous and greatly
improves nginx performance.

For 1 requests/connec

tcp: Make setsockopt(2) TCP_{NODELAY,NOPUSH,NOOPT,FASTKEEP} asynchronous

This makes all network system calls used by nginx asynchronous and greatly
improves nginx performance.

For 1 requests/connection (30K concurrent connections), this commit gives
12% improvement for 16KB web object (was 16.8Gbps, now 19.2Gbps, which
maxes out 2x10Ge). And it gives 19% improvement for 1KB web object (was
156Kreqs/s, now 186Kreqs/s).

show more ...

tcp: Fix bind races due to async close and random socket inital msgport

Bind for TCP sockets is now serialized by netisr0. Add comment about
this bind race.

Reported-by: Justin Sherrill

inet/inet6: Remove the v4-mapped address support

This greatly simplies the code (even the IPv4 code) and avoids all kinds
of possible port theft.

INPCB:
- Nuke IN6P_IPV6_V6ONLY, which is always on

inet/inet6: Remove the v4-mapped address support

This greatly simplies the code (even the IPv4 code) and avoids all kinds
of possible port theft.

INPCB:
- Nuke IN6P_IPV6_V6ONLY, which is always on after this commit.
- Change inp_vflag into inp_af (AF_INET or AF_INET6), since the socket
is either IPv6 or IPv4, but never both. Set inpcb.inp_af in
in_pcballoc() instead of in every pru_attach methods. Add INP_ISIPV4()
and INP_ISIPV6() macros to check inpcb family (socket family and
inpcb.inp_af are same).
- Nuke the convoluted code in in_pcbbind() and in6_pcbbind() which is used
to allow wildcard binding to accepting IPv4 connections on IPv6 wildcard
bound sockets.
- Nuke the code in in_pcblookup_pkthash() to match IPv4 faddr with IPv6
wildcard bound socket.
- Nuke in6_mapped_{peeraddr,sockaddr,savefaddr}(); use in6_{setpeeraddr,
setsockaddr,savefaddr}() directly.
- Nuke v4-mapped address convertion functions.
- Don't allow binding to v4-mapped address in in6_pcbind().
- Don't allow connecting to v4-mapped address in in6_pcbconnect().

TCP:
- Nuke the code in tcp_output() which takes care of the IP header TTL
setting for v4-mapped IPv6 socket.
- Don't allow binding to v4-mapped address (through in6_pcbbind()).
- Don't allow connecting to v4-mapped address and nuke the related code
(PRUC_NAMALLOC etc.).
- Nuke the code (PRUC_FALLBACK etc.) to fallback to IPv4 connection if
IPv6 connection fails, which is wrong.
- Nuke the code for v4-mapped IPv6 socket in tcp6_soport().

UDP:
- Nuke the code for v4-mapped IPv6 socket in udp_input() and udp_append().
- Don't allow binding to v4-mapped address (through in6_pcbbind()).
- Don't allow connecting to v4-mapped address.
- Don't allow sending datagrams to v4-mapped address and nuke the related
code in udp6_output().
- Nuke the code for v4-mapped IPv6 socket in udp6_disconnect()

RIP:
- Don't allow sending packets to v4-mapped address.
- Don't allow binding to v4-mapped address.
- Don't allow connecting to v4-mapped address.

Misc fixup:
- Don't force rip pru_attach method to return 0. If in_pcballoc() fails,
just return the error code.

show more ...

icmp: Asynchonize ctlinput/mtudisc/redirect processing

Supportive facilities added:
- Embed netmsg_ctlinput into mbuf (used by input ICMP datagram) so that
we don't need to do additional memory al

icmp: Asynchonize ctlinput/mtudisc/redirect processing

Supportive facilities added:
- Embed netmsg_ctlinput into mbuf (used by input ICMP datagram) so that
we don't need to do additional memory allocation to forward the
pr_ctlinput to all netisrs for certain types of ICMP datagrams.
- Add/Expose so_pr_ctlport().
- Return target netisr's cpuid from pr_ctlport. If the returned cpuid
equals to ncpus, it means that pr_ctlinput needs to be run in all
netisrs.
- Support 'direct' mode for TCP/UDP's pr_ctlinput. The caller of the
'direct' mode pr_ctlinput will do the message forwarding.

ICMP pr_ctlinput/mtudisc/redirect processing changes:
- For mtudisc and redirect we unconditionally forward the message
embedded in input ICMP datagram to all netisrs, and call pr_ctlinput
in 'direct' mode.
- For pr_ctlinput, we first call so_pr_ctlport() to see whether we need
to forward the message embedded in input ICMP datagram to all netisrs,
and act accordingly.
- Once pr_ctlinput/mtudisc/redirect processing is done, the message
embedded in the input ICMP datagram is forwarded back to the netisr0
to run rip_input().

After this commit, netisr0 will not be blocked for ICMP pr_ctlinput/
mtudisc/redirect processing. This also fixes the mtudisc (originally
only cpu0's routes get mtu updates).

show more ...

netmsg: ctlinput is proto message not a proto-user message

udp: Make udp pcbinfo and portinfo per-cpu; greatly improve performance

MAJOR CHANGES:

- Add token to protect pcbinfo's inpcb list and wildcard hash table.
Currently only udp per-cpu pcbinfo sets

udp: Make udp pcbinfo and portinfo per-cpu; greatly improve performance

MAJOR CHANGES:

- Add token to protect pcbinfo's inpcb list and wildcard hash table.
Currently only udp per-cpu pcbinfo sets this token. udp serializer
and netisr barrier are nuked.

o udp inpcb list:

Under most cases, udp inpcb list is operated in its owner netisr.
However, it is also accessed and modified (no effiective udp inpcb
will be unlinked though) in netisr0 to adjust multicast options if
one interface is to be detached. So protecting udp inpcb list
accessing and modification w/ token is necessary.

At udp inpcb detach time, the udp inpcb is first removed from the
udp inpcb list, then a message will go through all netisrs, which
makes sure that no netisrs are using or can find this udp inpcb
from the udp inpcb list. After all these, this udp inpcb is
destroyed in its owner netisr.

In netisrs, it is MP safe to find a udp inpcb from udp inpcb list,
then release the token and process the found udp inpcb.

In other threads, it is MP safe to find a udp inpcb from udp inpcb
list, then release the token and process the found udp inpcb in
non-blocking fashion.

See also the usage of inpcb marker.

o udp wildcard hash table:

On input path, udp wildcard hash table is searched in its owner
netisr. In order to ease implicit binding (bind during send),
connect after binding, and disconnect, udp inpcb are inserted
into and removed from other udp pcbinfos' wildcard hash table in
its owner netisr. Thus the udp wildcard hash table must be
protected w/ token.

At udp inpcb detach time, a message will go through all netisrs,
and this udp inpcb will be removed from the udp wildcard hash
table belonging to the current netisr. This makes sure that once
the current netisr runs the message handler, this udp inpcb will
not be used and be found in the current netisr. When the message
reaches the last netisr, this udp inpcb is redispatched to its
owner netisr to be destroyed.

In netisrs, it is MP safe to find a udp inpcb from udp wildcard
hash table, then release the token and process the found udp inpcb,
e.g. use udp inpcb found by in_pcblookuphash().

In other threads, it is MP safe to find a udp inpcb from udp
wildcard hash table, then release the token and process the found
udp inpcb in non-blocking fashion.

See also the usage of inpcb container marker.

o udp connect hash table:

It is lockless MP safe, and only accessed and modified in its owner
netisr.

- During inpcb iteration through inpcb list, use inpcb marker when
calling functions, which may block, e.g. in_pcbpurgeif0(), so the
inpcb iteration will not stop prematurely, if the inpcb being
processed is removed from the inpcb list.

- Use udp inpcb wildcard table and udp inpcb connect hash table to
dispatch input multicast and broadcast udp datagrams. Using udp inpcb
list could be time consume, since we need to check udp inpcb lists on
all cpus; and secondly, once udp inpcb has a local port, it will be in
either udp wildcard hash table or udp connect hash table.

Since the socket buffer operation on input path may block, inpcb
container marker is used when iterating inpcbs from udp inpcb wildcard
hash table. in_pcblookup_pkthash() is adjusted to skip inpcb
container marker.

- udp socket so_port is no longer fixed to netisr0 msgport
o Initial udp socket so_port is the current cpu's netisr msgport.
o Bound but unconnected udp socket so_port is selected according to
local port hash.
o Connected udp socket so_port is selected according to the udp hash,
i.e. laddr/faddr toeplitz hash (exception: multicast laddr or
multicast faddr, is hashed to netisr0).
o Multicast socket options are forced to be handled in netisr0, since
udp socket so_port may not be netisr0 msgport.

- In order to support asynchronized udp inpcb detach:
o EJUSTRETURN from pru_detach method now means protocol will call
sodiscard() and sofree() for soclose(). udp pru_detach method
returns EJUSTRETURN as of this commit.
o SS_ISCLOSING socket state is set before calling pru_detach method,
so protocol could avoid certain expensive, unnecessary or
disallowed operation in pru_disconnect or pru_detach method, e.g.
udp pru_disconnect method avoids putting udp inpcb back to udp
wildcard hash table, if SS_ISCLOSING is set.

MISC CHANGES:

- pcbinfo's cpu id must be set now; -1 is disallowed.
- udp pru_abort method should never be called; it panicks now.
- Restore traditional BSD behaviour, if unbound udp socket connect
fails: if local port of the udp socket has been selected, its inpcb
should be in wildcard hash table, i.e. the udp inpcb should be visible
on udp datagrams input path.
- Make sure multicast stuffs are adjusted only in netisr0 for inet6, if
one interface is about to be detached.

PERFORMANCE IMPROVEMENT:

For 'kq_connect_client -u' test, this commit gives 400% performance
improvement (31Kconns/s -> 160Kconns/s).

show more ...

tcp: Avoid unnecessary calling of in_pcbladdr() in tcp_connect()

If in_pcbladdr() was called and inp_laddr was saved, there would be
no need to call in_pcbladdr() again, after the inp was forwarded

tcp: Avoid unnecessary calling of in_pcbladdr() in tcp_connect()

If in_pcbladdr() was called and inp_laddr was saved, there would be
no need to call in_pcbladdr() again, after the inp was forwarded to
its owner netisr.

show more ...

udp: Dispatch UDP datagrams to the correct netisr to perform ip_output()

Redispatch UDP datagrams to the netisr, whose owner CPU matches the
UDP datagrams hash, to do the ip_output(). As measured b

udp: Dispatch UDP datagrams to the correct netisr to perform ip_output()

Redispatch UDP datagrams to the netisr, whose owner CPU matches the
UDP datagrams hash, to do the ip_output(). As measured bt KTR, the
udp_send() spends most of its time in ip_output().

To properly support this, following stuffs are added/changed:
- Add a network private lwkt message flag to indicate the so_port of
the netmsg should not be matched against the current netisr's msgport.
This flag is set when we redispatch the UDP datagrams to other netisrs.
- Don't use priority messages for netisr barrier, so at UDP socket close
time, UDP datagrams pending on other netisr msgport could be properly
sync'ed.

The UDP datagrams redispatch itself:
- If IP options are ever configured (supposely none), a copy of it will
be carried along w/ the UDP datagram to the target netisr. The copy
is made mainly because it is not safe to access the IP options of the
inpcb in netisr which does not own the inpcb.
(*) On the other hand accessing inpcb's multicast options is safe since
multicast UDP datagrams output and multicast options configuration all
happen in netisr0.
- Add nm_priv into netmsg_pru_send, which saves flags to be passed to
ip_output() for the UDP datagram. This does not changes the size of
netmsg_pru_send on x86_64 due to the implicit 4bytes padding.
- udp_addrcpu_pkt() is added to calculate the "real" CPU for the UDP
datagrams.
- Don't use inpcb route cache for redispatched UDP datagrams. Since:
o The cached route is usually not for the UDP datagrams' destination.
o Accessing inpcb route cache in the netisr, which is not the owner
of the inpcb is not safe.

On i7-3770 w/ 82599ES, this increases 18bytes UDP request/response
performance by ~19% (1.12M trasactions/s ---> 1.34M transactions/s)

This commit also makes lockless firewall state table doable, since input
and output of UDP datagrams, which have same hash, are running in the
same netisr now!

show more ...

tcp: Implement random initial msgport

After e368a6e95e2cd9556a3e0fc43167d2dcf3a8253f, it is found that the
left bottle neck of nonblocking TCP connect(2) performance is that all
socket(2) and initia

tcp: Implement random initial msgport

After e368a6e95e2cd9556a3e0fc43167d2dcf3a8253f, it is found that the
left bottle neck of nonblocking TCP connect(2) performance is that all
socket(2) and initial TCP connect operation (bind laddr and lport) are
all carried out in netisr0; CPU0 is 100% busy during test.

The idea of random initial msgport for TCP is that instead of using
netisr0's msgport as initial msgport, we could use any of the available
netisr msgport to carry out socket(2) and initial TCP connect operation.

Most of parts of TCP are already ready for random initial msgport, only
TCP pru_listen requires trivial modification to fix the socket msgport
to netisr0's msgport (which is required to perform global wild hashtable
updating).

As of this commit, the current CPU's netisr msgport will be selected as
TCP socket's initial msgport, if random initial msgport is enabled.

Sysctl node kern.ipc.rand_initport is added to disable this optimization.
It is enabled by default.

This commit improves both nonblocking TCP connect(2) and blocking TCP
connect(2) performance.

Nonblocking connect(2) performance measurement (i7-2600 w/ bnx(4)), using
tools/tools/netrate/accept_connect/kq_connect_client:

kq_connect_client -4 SERVADDR -p SERVPORT -i 8 -c 32 -l 30
(8 processes, each creates 32 connections simultaniously)

16 run average:
random initial msgport netisr0 msgport
263915.17 conns/s 220979.89 conns/s

This commit gives ~19% performance improvement for nonblocking connect(2)

Blocking connect(2) performance measurement (i7-2600 w/ bnx(4)), using
tools/tools/netrate/accept_connect/connect_client:

connect_client -4 SERVADDR -p SERVPORT -i 256 -l 30
(256 processes)

16 run average:
random initial msgport netisr0 msgport
240235.23 conns/s 198312.87 conns/s

This commit gives ~21% performance improvement for blocking connect(2)

show more ...

tcp: Implement asynchronous pru_connect

This is mainly used to improve TCP nonblocking connect(2) performance.

Before this commit the user space thread uses nonblocking connect(2)
will have to wait

tcp: Implement asynchronous pru_connect

This is mainly used to improve TCP nonblocking connect(2) performance.

Before this commit the user space thread uses nonblocking connect(2)
will have to wait for the netisr completes the SYN output. This could
be performance hit for nonblocking connect(2). First, the user space
thread is put into sleep, even if the connect(2) is nonblocking.
Second, it does not make too much sense for nonblocking connect(2) to
wait for the SYN output.

TCP's asynchronous pru_connect implementation will set ISCONNECTING
before dispatching netmsg to netisr0. The errors like EADDRNOTAVAIL,
i.e. out of local port space, will be notified through kevent(2) or
getsockopt(2) SOL_SOCKET/SO_ERROR.

NFS and other kernel code still use old synchronized pru_connect. This
commit only affects connect(2) syscall.

Sysctl node kern.ipc.soconnect_async is added to enable and disable
asynchronous pru_connect. It is enabled by default.

The performance measurement (i7-2600 w/ bnx(4)), using
tools/tools/netrate/accept_connect/kq_connect_client:

kq_connect_client -4 SERVADDR -p SERVPORT -i 8 -c 32 -l 30
(8 processes, each creates 32 connections simultaniously, run 30 secs)

16 runs average:

asynchronous pru_connect synchronized pru_connect
220979.89 conns/s 189106.88 conns/s

This commit gives ~16% performance improvement for nonblocking connect(2)

show more ...

uipc: Reference the originator thread if asynchronized pru_send is used

It is required for UDP asynchronized pru_send, which could access the
originator's thread.

netisr: Renaming, cpufn -> hashfn; no functional changes

Merge branch 'master' of git://git.dragonflybsd.org/dragonfly into wlan_serialize

arp: Embed netmsg_inarp in mbuf for asynchronized ARP input processing

- Embed netmsg_inarp, which records necessary states for routing table
updating and later ARP reply, into mbuf; this does not

arp: Embed netmsg_inarp in mbuf for asynchronized ARP input processing

- Embed netmsg_inarp, which records necessary states for routing table
updating and later ARP reply, into mbuf; this does not change mbuf
header size.
- If routing tables need updating upon ARP packet reception, the
netmsg_inarp embedded in the input ARP packet is sent asynchronizingly
to routing threads and the possible ARP reply is defered until all
routing tables are updated, i.e. the ARP packet is redispatched to
netisr0 for the ARP reply sending from the last routing thread.
- Remove no longer needed dedicated network threads.

Discussed-with: dillon@

show more ...

netmsg: Update comment

Merge branch 'master' of git://git.dragonflybsd.org/dragonfly into wlan_serialize

socket: Mark the asynchronous rcvd netmsg dead, when it is dropped

This is an attempt to fix the race that asynchronous rcvd netmsg
is queued to netisr after it is dropped in tcp_close().

Reported-

socket: Mark the asynchronous rcvd netmsg dead, when it is dropped

This is an attempt to fix the race that asynchronous rcvd netmsg
is queued to netisr after it is dropped in tcp_close().

Reported-by: ftigeot@

show more ...

Merge branches 'hammer2' and 'master' of ssh://crater.dragonflybsd.org/repository/git/dragonfly into hammer2

tcp: Implement asynchronized pru_rcvd

This mainly avoids extra scheduling cost on the reception path due to
lwkt_domsg(). lwkt_sendmsg() is now used to carry out TCP pru_rcvd.

Since TCP's pru_rcvd

tcp: Implement asynchronized pru_rcvd

This mainly avoids extra scheduling cost on the reception path due to
lwkt_domsg(). lwkt_sendmsg() is now used to carry out TCP pru_rcvd.

Since TCP's pru_rcvd could be batched, one pru_rcvd netmsg is embedded
into struct socket to avoid pru_rcvd netmsg allocation for each pru_rcvd,
and this netmsg will be used by lwkt_sendmsg(). Whether this embedded
pcu_rcvd netmsg should be sent or not is determined by its MSG_DONE bit.
Since user thread and netisr thread could be on different CPUs, the
embedded pru_rcvd netmsg's MSG_DONE bit is protected by a spinlock.

To cope with the following race that could drop window updates,
tcp_usr_rcvd() replies asynchronized rcvd netmsg before tcp_output():

netisr thread user thread

tcp_usr_rcvd() sorcvtcp()
{ {
tcp_output() :
: :
: sbunlinkmbuf()
: if (rcvd & MSG_DONE) (2)
: lwkt_sendmsg(rvcd)
: :
lwkt_replymsg(rcvd) (1)
}

At (2) window update is dropped, since rcvd netmsg is not replied yet at (1)

The result:
On i7-2600 (4C/8T, 3.4GHz):
32 parallel netperf -H 127.0.0.1 -t TCP_STREAM -P0 -l 30 (4 runs, unit: Mbps)

old 30253.88 30242.58 30162.55 30101.51
new 33962.74 33798.70 33499.92 33482.35

This gives ~12% performance improvement.

show more ...

pru_send: Allow non-NULL address parameter to be passed

Currently the passed in address is copied into a newly allocated
memory (grr, additional blocking kmalloc), and the PRUS_FREEADDR
will be set

pru_send: Allow non-NULL address parameter to be passed

Currently the passed in address is copied into a newly allocated
memory (grr, additional blocking kmalloc), and the PRUS_FREEADDR
will be set so that protocol thread could know when to free the
address.

Before this change netperf UDP_STREAM (unconnected socket) could
only do ~200Kpps (w/ -m 18), now it could do ~990Kpps (w/ -m 18).
This gives ~500% performance improvement for tiny UDP packet TX.
The improvement is not as good as the connected socket, which is
~600%, mainly because of the additional memory allocation for
the address. We _may_ further optimize out the address allocation.

show more ...

udp: Prepare for the asynchronized pru_send 2

Save the MSG_DONTROUTE in the netmsg instead of messing w/ socket.so_options

Initial import of binutils 2.22 on the new vendor branch

Future versions of binutils will also reside on this branch rather
than continuing to create new binutils branches for each new version.