Search — Nixt Server docs

Searching mail, whether with IMAP SEARCH, a JMAP app’s search box or Nixt Mail, is answered from a search index kept for each account. A search costs about what its answer costs, so it stays fast in a mailbox of hundreds of thousands of messages.

What a search matches

A search finds text anywhere in the part of the message it is asked about: the sender, recipients or subject, the message’s text (plain and HTML), or the whole message. It matches like this:

Any part of a word, address or number, not only whole words: voice finds invoice, and example finds ada@example.com.
Upper and lower case alike.
Chinese, Japanese and Korean text in sequences of two characters or more.
Accents exactly as typed: cafe doesn’t find café.

Inside attachments

A search of a message’s text also looks inside what it carries: “the contract Bob sent in March” is found by what the contract says, not only by its file name. The words of these attachments are read:

Kind	Files
Word processing	Word (`.docx`) and OpenDocument text (`.odt`)
Spreadsheets	Excel (`.xlsx`), OpenDocument spreadsheets (`.ods`) and CSV
Presentations	PowerPoint (`.pptx`) and OpenDocument presentations (`.odp`)
Plain text	`.txt` files, in whatever character set they declare
PDF	`.pdf` files with a text layer, page by page

Password-protected and damaged files are left unread, as are pictures, and a scanned PDF has no words to read; the rest of such a message is still searched. Attachments are read with strict limits on size and on how far a compressed file may expand, so a file built to exhaust the server is simply left unread.

Searches of fewer than three letters, and searches on flags, dates and sizes, still work. They are answered by reading the messages themselves.

The index only rules out messages that cannot match; every result is then checked against the message itself. So a search through the index gives exactly the results reading every message would.

Deleted mail that can still be got back is never shown in search results, and is found again as soon as it is put back.

Keeping the index

New mail is added to the index within seconds of arriving, in the background, so delivery never waits for it. Nothing needs setting up for that.

Mail that was already there before the index existed, such as mail from a server upgrade or a migration, is added by a build. Until it is added, searches still find it by reading it, so results are always complete; only speed improves as the build goes on.

The operator runs these on the server machine, as the versealx user:

sudo -u versealx versealx-server index status
sudo -u versealx versealx-server index build --tenant Acme --rate 50
sudo -u versealx versealx-server index verify --account ada@example.com
sudo -u versealx versealx-server index rebuild --account ada@example.com

Command	What it does
`status`	How far each organisation’s indexes have got, or one’s with `--tenant`.
`build`	Adds an organisation’s older mail to its accounts’ indexes, oldest first, at `--rate` messages a second (50 unless set, up to 10,000). It runs on the server in the background; `status` shows its progress.
`verify`	Checks a sample of an account’s messages against its index (`--sample`, how many), and starts the account’s index again if they disagree.
`rebuild`	Starts one account’s index again from nothing. Its searches read the messages meanwhile, so they stay complete.

These work beside a running server, and each is a line in the audit log. Only the server’s operator can run them; an organisation’s administrators cannot.

When the server is upgraded to a version that indexes more of each message (such as the words of attachments), each account’s index is rebuilt in the background by the server’s own housekeeping. Until an account’s is rebuilt, its searches read the messages, so they stay complete.

When an organisation is moved to another server, its indexes are not copied, since they are sealed with a key that stays behind. New mail is indexed where it lands, and index build there takes in the rest; until then its searches read the messages, so they stay complete.

Privacy

The index is stored encrypted with the organisation’s own key, like its mail. Without that key, it says nothing about which words are in anybody’s mail.

Watching it

The metrics endpoint reports:

Metric	What it shows
`vsx_search_seconds{protocol, answered}`	How long searches take, and whether the index (`index`) or reading every message (`scan`) answered them.
`vsx_search_index_lag_seconds`	How long new mail waits before it is in the index.
`vsx_search_index_terms`	How many index entries each message makes.
`vsx_search_index_failures_total`	Messages that could not be indexed. They are still found by reading them.
`vsx_search_attachments_skipped_total{reason}`	Attachments whose words were not read, by why: `encrypted`, `corrupt`, `too_large`, or a limit a file met.

Over the API

For the operator:

Route	What it does
`GET /api/v1/search-index`	How far each organisation’s indexes have got, or one’s with `tenant`.
`POST /api/v1/search-index/build`	Starts a build of an organisation’s older mail, with `tenant` and `rate`.
`POST /api/v1/search-index/rebuild`	Starts one account’s index again, with `account`.