fix some typos; courtesy of codespell; ok gilles@

add missing include of time.h

spotted after a report on OpenSMTPD-portable. While here include
sys/time.h in smtpd.h, as noted in event_init(3), since it includes
event.h.

ok millert@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

switch smtpd to new grammar

ok eric@

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

show more ...

Implement smtpctl discover <evpid|msgid>.

discover subcommand schedules envelopes manually moved to the queue.
It triggers a queue walk searching for envelopes with the given id,
schedules them and

Implement smtpctl discover <evpid|msgid>.

discover subcommand schedules envelopes manually moved to the queue.
It triggers a queue walk searching for envelopes with the given id,
schedules them and informs the user number of envelopes scheduled.
Admins no longer would need to restart the daemon to discover
manually moved messages.

Ok gilles@

show more ...

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable shou

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

show more ...

Improve the scheduler, better and simpler.

- Get rid of the scheduler_batch structure. The scheduler can now return
envelopes of different types in a single run, interlacing them to avoid
batch

Improve the scheduler, better and simpler.

- Get rid of the scheduler_batch structure. The scheduler can now return
envelopes of different types in a single run, interlacing them to avoid
batch effects.

- Ask for an acknowledgement from the queue when removing or expiring
an envelope to benefit from the inflight envelope limitation mechanism.
This ensures that the scheduler always keeps sending envelopes at a rate
that the queue can sustain in all cases.

- Limit the number of envelopes in a holdq. When a holdq is full,
new envelopes are put back in the pending queue instead, with a
shorter retry time.

- Plumbing for proc-ified schedulers.

imsg version bump. smtpctl stop before updating.

ok gilles@

show more ...

fix typos.

ok jmc@

(void) cast strlcpy/strlcat/snprintf calls that cannot truncate (and would
be harmless in this case if they could)

internal improvements and cleanups

- get rid of the whole penalty thing for failed envelopes in the mta and scheduler.
- do not disable routes on smtp errors
- try to schedule all types of envelopes

internal improvements and cleanups

- get rid of the whole penalty thing for failed envelopes in the mta and scheduler.
- do not disable routes on smtp errors
- try to schedule all types of envelopes on each scheduler frame.

show more ...

bcopy -> memmove
bzero -> memset

When a relay fails, let the scheduler update all envelopes in the
holdq as if they tempfailed.

schedule in O(log n)

Rework the mda and scheduler to use the holdq mechanism instead of
tempfail for limiting the number of pending deliveries to the same
user. This allows to reach optimal delivery time even in case of

Rework the mda and scheduler to use the holdq mechanism instead of
tempfail for limiting the number of pending deliveries to the same
user. This allows to reach optimal delivery time even in case of
burst, while keeping the number of inflight envelopes low.

show more ...

Implement a feedback mechanism which allows the mta to "hold" envelopes
in the scheduler when it has too many tasks for a given relay. The
envelopes are put on a wait queue, and are not scheduled ag

Implement a feedback mechanism which allows the mta to "hold" envelopes
in the scheduler when it has too many tasks for a given relay. The
envelopes are put on a wait queue, and are not scheduled again until
the mta "releases" some envelopes from that queue.

It prevents from having too many inflight envelopes, which are out of reach
for the admin.

show more ...

If the admin issues a remove request for an envelope which is inflight, do not
ignore the request. Instead defer the removal until the envelope returns to the
scheduler. Simplify code by a great deal

If the admin issues a remove request for an envelope which is inflight, do not
ignore the request. Instead defer the removal until the envelope returns to the
scheduler. Simplify code by a great deal in the process.

show more ...

Implement a scheduler_proc backend

scheduler improvements:
- implement suspend/resume scheduling for individual envelopes or message,
with the associated smtpctl commands.
- allow the mta to request immediate scheduling of an envelo

scheduler improvements:
- implement suspend/resume scheduling for individual envelopes or message,
with the associated smtpctl commands.
- allow the mta to request immediate scheduling of an envelope.
- on temporary failures a penalty can be given to further delay the next try.

show more ...

sync with OpenSMTPD 5.3.2

ok gilles@

When getting the next batch of envelope to schedule, use an array to
store envelope ids, rather than a dynamic list.

ok gilles@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of tem

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

show more ...

Allow "smtpctl show queue" to run in "online" mode if the smtpd server
is running. The scheduler sends the runtime state of each envelope to
the queue process which loads the envelope, fills the run

Allow "smtpctl show queue" to run in "online" mode if the smtpd server
is running. The scheduler sends the runtime state of each envelope to
the queue process which loads the envelope, fills the runtime bits and
sends the envelope back to the client. Iteration over the envelope set
happens in small chunks to make the request interruptible and to allow
the server to keep doing its job in the meantime.

Adpat "smtpctl schedule-all" to schedule the messages one by one using
the same iteration mechanism.

Document "smtpctl monitor" and "smtpctl show queue".

ok gilles@

show more ...