Sending and delivery

Mail leaves Nixt Server in two stages. First a signed-in person’s app hands a message to the server over SMTP submission or JMAP, and the server puts it on the queue. Then the relay role works through the queue: it delivers local recipients into their mailboxes and every other recipient to another server, retrying for up to five days.

Submission

People’s apps submit mail on port 587 with STARTTLS, or port 465 with TLS from the start. See Connecting mail apps for app settings and Signing in for authentication.

When a message is submitted, the server:

1. Checks who is sending. The envelope sender (MAIL FROM) must be one of the account’s addresses, or the address of a group it belongs to — or empty. The address in the From header must pass the same check.
2. Checks the domain may send outward. If the sender’s domain is not yet proved to belong to the tenant, recipients outside this server are refused. See Domain verification.
3. Filters it. The attachment policy and any scanner or milters run as for incoming mail. See Spam and malware filtering.
4. Completes it. Adds Message-ID and Date when the app left them out, and a Received header.
5. Signs it with every active DKIM key of the envelope sender’s domain — normally one RSA and one Ed25519 signature. A domain with no keys sends unsigned mail and the log says no DKIM key; sending unsigned.
6. Queues it, writing the message and its queue entry in one transaction, and replies 250 2.0.0 queued as <id>.

Submission replies

Sending a message that is already in a mailbox (BURL)

On submission, the server offers BURL imap. An app can send a draft or forward a message that is already stored by giving the server a signed IMAP URL instead of uploading the message again. The URL must be a submit+ URL for the signed-in account. This saves uploading large attachments twice.

Sending over JMAP

JMAP apps send with EmailSubmission/set, naming a message saved in the account, usually in Drafts. When no envelope is given, the message goes to its To, Cc and Bcc addresses from its Sender or From. The app can move the message to Sent in the same request. The message joins the same queue as submitted mail.

Sending later

A person can schedule a message to go at a time, up to 30 days ahead, from any app that supports it: “send tomorrow at 8”. The server sends it at that time, so the app does not need to be open then.

• Over SMTP, the submission ports offer FUTURERELEASE (RFC 4865). The app adds HOLDFOR=<seconds> or HOLDUNTIL=<UTC date-time> to MAIL FROM.
• Over JMAP, the session’s maxDelayedSend says how far ahead a message may go. The app puts HOLDFOR or HOLDUNTIL in envelope.mailFrom.parameters of EmailSubmission/set, and the answer’s sendAt says when it will go. The message waits in the person’s Scheduled folder, which every app sees.

Every check a send makes is made when the message is scheduled: who is sending, the sending limits and the outbound filter. So a message scheduled today does not fail next week for a reason known today. It then waits on the queue until its time, and its message trace says over submission, to go at <time>.

Until its time, a JMAP app can cancel it by setting its undoStatus to canceled. The message goes back to Drafts and is never sent. EmailSubmission/get and /query list what the person has scheduled, and undoStatus says whether each is still pending, final or canceled.

At its time the message is sent like any other. A message scheduled over JMAP is filed in Sent then, marked read.

A message goes at its time only while the person who scheduled it could still sign in. If the account was suspended or removed, the organisation suspended, or the account signed out everywhere after the message was scheduled (as a password reset and securing an account both do), the message is not sent. It is held for review instead, where the administrators send it or delete it. Deleted, a JMAP message goes back to the person’s Drafts.

A HOLDUNTIL time already past sends the message at once. Over JMAP the same mistakes are refused as invalidProperties naming envelope.

Devices that cannot sign in

A printer that scans to email, an alarm panel or an appliance that sends a nightly report often cannot hold a password. Name it in [[relay_devices]] and it may send without signing in, on port 25 or the submission ports, from the addresses you give it — and only as its own sender, to the recipients you name, at a rate of its own:

[[relay_devices]]
name = "scanner-floor-2"
from = ["192.0.2.40", "192.0.2.48/29"]
sender = "scanner@example.com"
recipients = ["@example.com"]
messages_per_hour = 100

Its mail then takes the path a signed-in person’s does: the outbound filter, the domain’s DKIM signature and the queue. The received step of its message trace says relayed for the device scanner-floor-2 at 192.0.2.40, which signs in as nobody. A device that signs in on the submission ports is treated as whoever it signs in as.

The queue

Every accepted message — incoming or outgoing — goes on the transport queue. The relay role looks at it every second.

How the route is chosen

For each recipient domain, in this order:

1. A domain hosted on this server: delivered locally.
2. A matching [outbound.transport] entry: sent to that relay.
3. [outbound.relay], when there is one: sent to the smart host.
4. Otherwise: the domain’s MX hosts, or its address records when it has no MX.

DNS answers are validated with DNSSEC. Addresses that are not globally routable — private, loopback, link-local and similar — are skipped unless [outbound] allow_private_addresses is on, and the log says address is not public; skipped.

Securing each hop

Before connecting to a destination’s MX host, the server decides how strictly the connection must be protected, strongest first:

When an opportunistic TLS handshake fails, the delivery is deferred rather than retried without encryption, unless every tenant with mail on that connection has set outbound.plaintext_retry to after-failed-handshake.

TLS agreements with partners

Some partners, such as a payroll bureau, a bank or a law firm, agree with you that mail between you is always protected. Record the agreement, and mail from your organisation to that partner’s domain is held to it:

An agreement only ever adds protection. Where the partner publishes DANE or MTA-STS, those still apply, and a pinned key is checked besides. A partner whose servers can’t meet the agreement has its mail wait in the queue, with a reason that starts TLS agreement with <domain>:, until they can or the retry schedule runs out. Such mail is never retried unencrypted, whatever outbound.plaintext_retry says. An agreement is your organisation’s alone: other organisations on the same server send to the partner as they otherwise would.

An agreement can cover mail from the partner too. With Mail from it must be: Encrypted (inbound: require), the partner’s mail that arrives unencrypted is refused with 530 5.7.0 Must issue a STARTTLS command first, and the partner’s server tries again over TLS. Other senders are unaffected, and so are bounces, which name no sender to match.

On the console, the TLS agreements card on Settings lists them, with Add and Remove. From the command line:

versealx-server admin tls list
versealx-server admin tls add payroll.example --out verify --note "Payroll bureau"
versealx-server admin tls add bank.example --out pin --pin 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=
versealx-server admin tls add lawfirm.example --out verify --inbound require
versealx-server admin tls remove payroll.example

Through the API, GET and PUT /tenants/{tenant}/tls-agreements read and replace the whole list against its version. Every change is in the audit log.

MTA-STS policies are fetched over HTTPS from https://mta-sts.<domain>/.well-known/mta-sts.txt and kept for the policy’s max_age. A policy that arrives cut short is not used.

A relay’s connection follows its tls setting instead: required and implicit verify the certificate against the public roots and [outbound] trust; opportunistic does not.

Talking to the next hop

• The message is sent in one BDAT chunk when the other server offers CHUNKING, and with DATA otherwise.
• Mail your people send that asks for SMTPUTF8 only because of its headers, with every address in plain ASCII (a subject or a name in another script), has those headers written as encoded words when it is submitted, before it is signed. It then reaches any server, and every mail app shows the same text. A message with an address that isn’t ASCII still needs SMTPUTF8.
• If a message that needs SMTPUTF8 meets a next hop without it, and every address is ASCII, it is converted once the same way: its header fields written as encoded words, signed again in its From: domain’s name where your organisation had signed it, and sent again at once. A message with an address that isn’t ASCII, a field that can’t be written so, or a signature the server can’t make again fails with 550 5.6.7 next hop does not support SMTPUTF8.
• A message that needs BINARYMIME is never converted, because that would rewrite its MIME structure. If the next hop lacks it, the recipient fails with 550 5.6.1 next hop does not support BINARYMIME, ….
• A message larger than the next hop advertises fails with 552 5.3.4 message larger than the next hop's limit of <n> bytes.
• A permanent refusal (a 5xx reply) fails that recipient; a temporary one (4xx) is retried.

Retries

A recipient that could not be delivered is tried again on this schedule, counted from the previous attempt:

The schedule is measured from when the message arrived, so a restart changes nothing. Five days after arrival the last attempt is made and any recipient still not delivered fails with 451 4.4.7 delivery time expired; last error: <the last reply>.

Some failures are not retried:

And some are always retried:

Pacing

Each destination has a ceiling on connections at once, messages per connection, and messages and recipients a minute; see Configuration for the built-in values and Runtime settings to change them without a restart.

• A message whose destination is at its ceiling waits 5 seconds and tries again. Waiting for pacing never uses up one of the message’s retries.
• Nodes sharing a store share each destination’s allowance, so three nodes do not send three times as fast.
• A connection a stopped node was counted as holding is forgotten after 10 minutes.

Pacing the big providers

Mail to a domain whose mail is handled by Google, Microsoft, Yahoo (which also handles AOL) or Apple’s iCloud is paced as that provider, not as the domain alone. A company whose mail Google hosts shares Gmail’s pace, since it is the same receiver. The provider is found from the domain’s most preferred MX host, remembered for an hour. Other domains are paced by their own name, as before. The existing destination settings name a provider by its domain, such as outbound.destination.google.com.*.

When a provider asks the server to slow down, the server slows down for everybody’s mail to that provider, not just the one message:

• What counts as “slow down”: a temporary 421 or 451 reply with 4.7.28 (Gmail’s rate limit), 4.4.5 (the receiver is congested), Microsoft’s throttling codes, or 4.7.0 saying to try again later or naming a rate or reputation limit. Replies about authentication, or about one mailbox receiving too fast, are retried as usual and slow nothing down.
• What the server does: it halves its pace to that provider, never below an eighth, and waits before sending there again. The wait is 1 minute, doubling each time the provider asks again, up to 32 minutes. The provider’s other MX hosts are not tried meanwhile, since they are the same receiver.
• How it recovers: as the provider accepts mail again, the pace climbs back a little with each message accepted, until it is back to full.

Nodes sharing a store share each provider’s pace, and a node that restarts starts at the pace it had.

To see each provider’s pace and whether it is asking the server to slow down, open the queue page on the console (Sending pace), or:

vsx admin queue pacing

When a provider keeps the server slowed for an hour, the operator gets the provider-throttling alert (see Monitoring). An hour of throttling usually means the server’s reputation with that provider needs looking at: its authentication records, its sending volume, or a compromised account sending spam.

The metrics vsx_outbound_pace_percent{provider} and vsx_outbound_throttled_replies{provider} show the same for each named provider.

Addresses that don’t exist

When a remote server says a mailbox doesn’t exist — refusing the recipient itself with 550 5.1.1 or words to that effect — for two different messages, an hour or more apart within a week, the address goes on the organisation’s suppression list for ninety days. Mail to it is then refused at once when somebody sends it, with 550 5.1.1, instead of going out and bouncing again. Receivers count a sender that keeps writing to addresses that don’t exist against it, so this keeps the organisation’s mail welcome.

Only a refusal of the recipient counts. A server refusing the sender, or refusing a message after it was sent, says nothing about any one mailbox, and a message tried twice counts once. An address with an international domain is one address however it is written.

Every permanent refusal is also counted by day and by kind — mailbox doesn’t exist, mailbox full, refused by the receiver’s policy, refused for the sender’s reputation — whether or not it suppresses anything.

Bounces that come back later. Some servers accept a message and only later send back a delivery report saying it failed. The server reads such a report when it arrives at one of the organisation’s addresses, matches it to the message it really sent to that recipient, and counts its failure the same way. A late “mailbox doesn’t exist” for mailing-list mail counts towards the suppression list like a refusal. A report that names a message this server never sent to that recipient changes nothing, and every report is still delivered as ordinary mail.

Seeing and changing the list

Organisation administrators and auditors see the list; administrators take an address off, for example when its mailbox exists again. Taking it off forgets its bounces too, so one more doesn’t put it straight back.

• Console: Settings › Suppressed addresses, with Take off beside each.

• Command line:

  versealx-server admin suppression list
  versealx-server admin suppression remove old.colleague@partner.example

• API: GET /api/v1/tenants/{tenant}/suppression and DELETE /api/v1/tenants/{tenant}/suppression/{address}.

An address comes off by itself ninety days after its last bounce.

Complaints from feedback loops

Several large mail providers run a feedback loop: when one of their users marks your mail as spam, they send you a report of it (RFC 5965). Each domain on the server takes those reports at fbl@ the domain, whether or not anybody has that mailbox, and reads them as they arrive.

1. Enrol with each provider’s feedback loop, giving fbl@ one of your domains as the address to report to.

2. Name the address each loop sends its reports from, exactly as the provider gives it, in the server’s runtime settings:

   vsx admin patch tenants/0/settings 'deliverability.feedback_senders=["feedback@loop.provider.example"]'

A report counts only when it comes from one of those addresses and passes DMARC for that address’s domain. Anybody can write a report, so one from anywhere else is delivered to fbl@ and counts for nothing. The list is empty until you fill it.

A report is about the person whose address is the one From: of the message complained about, on mail signed by one of the organisation’s domains. Each complaint is kept for ninety days. When the feedback loops complain about one person’s mail 5 times in an hour, that person’s sending outside the organisation is held for review, and the organisation’s administrators are told. A report about mail the organisation did not send, or about nobody it has, holds nobody.

When the complaint is about a message one of the organisation’s mailing lists sent, and the person who complained is a member from outside the organisation, they are taken off the list, as if they had unsubscribed, and the audit log says so. Members inside the organisation are never taken off.

Feedback-ID for Gmail

Gmail’s Postmaster Tools reports how often recipients mark mail as spam, separately for each Feedback-ID value on mail your domain signs. The server adds Feedback-ID: <account number>:<organisation number>:<stream>:versealx before it signs, where the stream is list, notice or person. It holds numbers only, never a name or address. A Feedback-ID the author wrote is always replaced.

The organisation’s runtime setting deliverability.feedback_id decides which mail carries it: bulk, the default, for mailing lists and the server’s notices; all for people’s own mail too; or off.

Seeing the complaints

Organisation administrators and auditors see the complaints of the last 1 to 90 days, counted by person, most first, and the newest of them.

• Console: Settings › Complaints, below Suppressed addresses.

• Command line:

  versealx-server admin complaints show --days 30

• API: GET /api/v1/tenants/{tenant}/complaints?days=30.

An organisation’s sending ceilings

The operator can cap what one organisation sends outside itself, so that one organisation can’t damage the server’s reputation for everybody on it. Each is a most, and none is set until the operator sets it:

A message past a ceiling is deferred with 451 4.7.1, and goes once the hour or the day moves on.

Complaints. Each organisation also has a complaint ceiling, ceilings.complaints_per_thousand, a runtime setting: the complaints about its mail in the last day, per thousand messages it sent outside that day, counted once it has sent at least a thousand. Past it, mail to outside recipients is deferred with 451 4.7.1, saying recipients have complained about the organisation’s mail more than its ceiling allows. It is 1 unless the organisation sets it (0 to 1000); 0 turns it off. Mail between the organisation’s own people is never counted, and nor is a message the filter refuses or a rule holds.

Only the operator sets them, over the local socket:

versealx-server admin put tenants/3/ceilings messagesPerHour=500 messagesPerDay=5000 distinctOutsidePerDay=2000
versealx-server admin get tenants/3/ceilings

get shows the ceilings and where the organisation stands against them now. Setting them all to 0 takes them away. The organisation’s own administrators neither see nor change them.

Probation for new organisations

An organisation created on the server starts on probation: for its first 30 days it sends under low ceilings, whatever its own are, and the lower of the two always applies. Organisations that were already on the server, the first one made when the server was set up, and sandboxes are never put on probation.

Probation ends early, from day 7, once at least 50 outside recipients have been sent to and under 5% of them bounced. At day 30 it ends unless more than 5% bounced, in which case it stays and the operator gets the probation-kept alert. A message past a probation ceiling is deferred with 451 4.7.1, saying the organisation is new and on probation.

The operator can end probation at any time, or put an organisation on it, which then lasts until the operator ends it. Both are recorded in the audit log:

versealx-server admin tenant 3 probation show
versealx-server admin tenant 3 probation off
versealx-server admin tenant 3 probation on

The organisation’s administrators see on the organisation’s page whether it is on probation, how its bounces stand, and when probation ends.

Delivery status notifications

When the server gives up on a recipient, or a message has been waiting a long time, it tells the sender with a standard delivery status notification.

• Notifications come from Mail Delivery System <MAILER-DAEMON@mail.example.com> with an empty envelope sender and Auto-Submitted: auto-replied.
• The first part explains in words, for example Your message could not be delivered to one or more recipients. It is attached below. The second is the machine-readable status of each recipient. The third is the original message.
• The whole original is attached when it is 100 KiB or smaller and the sender did not ask for headers only; otherwise only its headers are.
• By default a sender hears about failures and delays. An app can ask for other notifications, or none, with the SMTP NOTIFY and RET parameters.
• A notification about an internationalised address uses the internationalised report format when the sender’s own address is internationalised, and writes the address in an escaped ASCII form otherwise.

No notification is sent:

• about a notification, so two servers can never bounce mail back and forth;
• to a sender whose message arrived from another server and passed neither SPF nor DKIM, so forged senders are not sent mail they never wrote.

Sending through a relay

A smart host takes all mail for other domains:

[outbound.relay]
host = "smtp.relay.example.net"
port = 587
tls = "required"
user = "postmaster@example.com"
password = "${VERSEALX_RELAY_PASSWORD}"

1. Add the table to the configuration, or run init with --relay smtp.relay.example.net:587 when setting up.

2. Make the password available to the service as the environment variable you named, for example in a systemd drop-in created with sudo systemctl edit versealx-server:

   [Service]
   Environment=VERSEALX_RELAY_PASSWORD=YOUR_PASSWORD

3. Add the relay to your SPF record: v=spf1 mx include:<the relay provider's SPF domain> -all.

4. Restart the service and run doctor, which checks that SPF includes the relay.

The server authenticates with AUTH PLAIN, falling back to LOGIN, and never sends the credentials over an unencrypted connection. Incoming mail is not affected.

To send only some destinations through a relay, use [outbound.transport."<domain>"] with the same keys; see Configuration.

A relay that cannot be resolved fails the attempt with relay <host>: <reason>, retried like any other failure.

Forwarding with Sieve

When a person’s Sieve script redirects a message, the forwarded copy goes on the queue like any other message; a redirect outside the organisation goes only once the organisation allows it (see Forwarding outside the organisation). Its envelope sender is rewritten to an address in the parent domain of the server’s host name — for mail.example.com, an address at example.com — so SPF at the destination checks your domain rather than the original sender’s. Make sure that domain’s SPF record allows this server.

Local delivery

The deliver role puts a message into each local recipient’s mailboxes; see Receiving mail. If the store refuses a delivery — for a full disk, for example — it is retried, and after five days the sender is told 552 5.2.2 delivery gave up: <reason>.

Watching outgoing mail

See Message trace and Monitoring.