Add documentation.

FossilOrigin-Name: 3f015df490d1df1dd9800a0db1b0aa28d8af6ee688e7a09ba21b8b1e2f58c465

README.md

@@ -2,14 +2,220 @@
 Sieb
 ====
 
+Streamlined mail filtering and delivery.
+
+
 What's this?
 ------------
 
+I used procmail for nearly 25 years.  In time, I migrated to maildrop.  Had some
+minor frustrations, and finally just decided to scratch my own itch.  Maybe you're
+itchy too?  This might scratch it. It might not.  Read on to find out.
+
+Sieb is nowhere near as functional as procmail or maildrop, but it is geared
+specifically for how -I- used those tools, which in 25 years, definitely covers
+my primary use case - and no others.
+
+Here's what Sieb is good at:
+
+  - Matching email on arbitrary headers, and delivering to Maildir.
+  - Piping messages through "filters" dedicated for that purpose - formail,
+    spamprobe, bogofilter, etc
+  - Caring about your mail: any fatal exception throws the message back to the MTA queue
+  - Runtime exceptions deliver the message to the default `$HOME/Maildir`
+  - Efficiency:  Written in Nim for speed, deliveries on modern hardware generally
+    are under 10ms and consume around 16k of memory
+
+Other behaviors to be aware of -- to be considered features, or perhaps reasons
+to steer clear:
+
+  - Designed for use with Qmail.  Presumably you can use it with other MTAs,
+    but I haven't tested it.  YMMV.
+  - Only supports Maildir.  I haven't used an MBox since 1998 and I don't intend to.
+  - There is no mail forwarding logic.  Just filtering and delivery.
+  - Deliveries to Maildir are always presumed to be relative under `$HOME/Maildir`,
+    as 'child' Maildirs. This is for interaction with Maildir-aware environments
+    such as Dovecot, and keeps all your email to one tidy spot.
+  - Maildir delivery paths are always created automatically if not already
+	present, no need to worry about maildirmake nonsense.
+  - Sieb only filters email on RFC2822 headers.  SMTP bodies are not considered.
+  - There are roughly two "phases", both are optional - matching before an
+	external filter, and matching afterwards.  This is for delivery before and
+	after an out-of-band spam filter.
+  - Sieb can be used as a system-wide default delivery agent, for site specific
+    delivery instructions.  Users can override the side-wide settings, following
+	the Qmail mantra.
+  - There is no custom programming "language" to learn - no conditionals, no
+    variables.  Just a YAML file expressing delivery instructions.
+
 
 Installation
 ------------
 
+You can check out the current development source with Fossil via its
+[home repo](https://code.martini.nu/fossil/sieb), or with Git at its
+[project mirror](https://github.com/mahlonsmith/sieb).
+
+Alternatively, you can download the latest version from:
+
+    https://code.martini.nu/fossil/sieb/uv/sieb-latest.tar.gz
+
+
+**NOTE**: As of Nim v0.16.4, the Nimble package manager is still at v0.13.1.
+Sieb requires >= v0.14 of Nimble, you may need to update it first via:
+
+    % nimble install nimble
+
+
+With the [nim](https://nim-lang.org/) environment installed and the Sieb
+repository cloned, simply type:
+
+    % make
+
+That will result in an optimized `sieb` binary.  Put it into your $PATH.
+
+
+Then simply instruct Qmail to deliver to Sieb - either site-wide, or for
+yourself via a `.qmail` file as such:
+
+    | /path/to/sieb
+
+
+By default, Sieb delivers to `$HOME/Maildir` as if it wasn't there at all.  In
+order to control behavior, you'll need a configuration file.  You can generate a
+commented example file and put it where Sieb can find it as such:
+
+    % mkdir -p ~/.config/sieb && sieb -g > ~/.config/sieb/config.yml
+
 
 Configuration
 -------------
 
+Sieb uses a YAML file to describe filtering and delivery behavior.
+
+It looks for a file in the following locations:
+
+      - ~/.config/sieb/config.yml
+      - /usr/local/etc/sieb/config.yml
+      - /etc/sieb/config.yml
+
+You can also specify a file via the `-c (--conf)` flag:
+
+    % sieb --conf=/path/to/config.yml
+
+Without a config file, Sieb is transparent.
+
+
+There are only three things to know for Sieb config -- rules, filters, and
+delivery destinations.
+
+
+### Filters
+
+A filter is a pipe to an external program.  It should accept input on stdin, and
+emit on stdout.
+
+You can have any number of filters, either globally, or specific to a successful
+match.
+
+Filters are expressed as a YAML array, so you don't need to worry about escaping
+shell quotes, for example, for complex arguments.
+
+An example global filter chain that performs spam categorization, then adds a
+custom header:
+
+    filter:
+      - [ bogofilter, -uep ]
+      - [ reformail, -A, "X-Sieb: Processed!" ]
+
+
+### Delivery Destination
+
+Simply, the name of the child maildir to put a matched email into.  This is
+always relative to `$HOME/Maildir`:
+
+    deliver: .freebsd
+
+In the above example, a matching rule would deliver the message to:
+
+    ~/Maildir/.freebsd/new/
+
+In the absence of a delivery instruction, mail is delivered to `$HOME/Maildir`.
+
+
+### Rules
+
+A rule is the workhorse, combining filters and delivery destinations.  They are
+expressed as an array of YAML key/val pairs. Each key represents a header, each
+value is a [PCRE-style](http://pcre.org/current/doc/html/pcre2pattern.html)
+regular expression to test the header against.  If there are multiple headers
+listed for a rule, all must match.  Headers (and the regular expressions) are
+compared case-insensitively.
+
+Rules are executed top down, first match wins.
+
+    rules:
+      - 
+        match:
+          subject: whatever
+        filter:
+          - [ reformail, -A, "X-Sieb: Matched!" ]
+        deliver: .whatever-mail
+
+There is a special key called "TO", that matches both "To:" and "Cc:" headers
+simultaneously for convenience.  Keep in mind that mail addresses can include
+quotes, real names, etc.  Use greedy matching!
+
+      - 
+        match:
+          TO: .*freebsd-questions@FreeBSD.org.*
+        deliver: .freebsd-lists
+
+
+The YAML parser is (extremely) strict.  If something doesn't seem to be working,
+it's likely due to an error in your YAML - which brings me to...
+
+
+Debugging
+---------
+
+You can use the `-l (--log)` flag to instruct Sieb to write out what it is
+doing, along with any errors along the way.  Note that the logfile must remain
+locked during delivery, so it can slow simultaneous deliveries if you've got a
+busy incoming mailbox.
+
+The logfile is always written under `$HOME/Maildir`, so `--log=sieb.log` ends up
+at:
+
+    ~/Maildir/sieb.log
+
+If you're having a particularly hard time filtering a piece of mail, you can
+feed it into sieb directly with the `--debug` flag, and the same information
+will be emitted to stdout.  This is useful both when diagnosing a filter, and
+when trying out a new one without affecting current mail delivery.  You'll
+probably want to avoid this flag in production, as it'll likely fill your mail
+logs with garbage.  Sieb is silent by default.
+
+    % sieb -d < test-message.txt
+
+    2023-07-01T01:50:22Z ------------------------------------------------------------------
+    Using configuration at: config.yml
+    Opening new message at:
+      /home/mahlon/Maildir/tmp/1688176222.263185.1583965.1.kazak
+    Wrote 9623 bytes
+    Parsed message headers.
+    Message-ID is "<a111b6a7-0bf8-d3b9-9611-a7fbf36635b3@artem.ru>"
+    Evaluating rule...
+     checking header "list-id"
+        match on ".*freebsd-questions.*"
+    Rule match!
+    Delivered message to:
+      /home/mahlon/Maildir/.freebsd/new/1688176222.263185.1583965.1.kazak
+    Completed in 1.14ms, using 18.02Kb
+
+
+Reporting Issues
+----------------
+
+Please report any issues [here](https://code.martini.nu/fossil/sieb/tktnew).
+

config.yml

@@ -1,53 +0,0 @@
-#
-# Example Sieb configuration file.
-#
-
-
-# Rules tried before global filtering.
-#
-early_rules:
-  -
-    match:
-      TO: ahlon@(laika|ravn|martini)
-    filter:
-      - [ reformail, -A, "X-Sieb: This matched." ]
-    deliver: .whatever
-
-## Filter message before additional rules.
-filter:
-  - [ reformail, -A, "X-Sieb: Processed!" ]
-
-## Ordered, top down, first match wins.
-## Headers are lowercased.  Multiple matches are AND'ed.
-##
-## Delivery default is ~/Maildir, any set value is an auto-created maildir under
-## that path.
-##
-rules:
-  -
-    match:
-      x-one: global
-  -
-    match:
-      Subject: .*\s+corepacket\s+.*
-    deliver: .balls
-    filter:
-      - [ reformail, -A, "X-Sieb: Boom!" ]
-  -
-    match:
-      x-three: global
-
-
-  # # Magic "TO" which means To: OR Cc:
-  # -
-  #   headers:
-  #     TO: regexp
-  #   deliver: .whereever
-   
-  # # Filter message through reformail, then deliver to ~/Maildir.
-  # -
-  #   headers:
-  #     x-what: fuckery
-  #     filter: reformail ...
-
-

src/lib/logging.nim

@@ -12,6 +12,7 @@ import
     std/monotimes,
     std/os,
     std/posix,
+    std/strutils,
     std/times
 
 
@@ -39,13 +40,16 @@ proc createLogger*( parentdir: string, filename: string ): void =
     ## Get in line to open a write lock to the configured logfile at +path+.
     ## This will block until it can get an exclusive lock.
     let path     = joinPath( parentdir, filename )
-    logger       = Logger()
+    try:
-    logger.fh    = path.open( fmAppend )
+        logger       = Logger()
-    logger.start = getMonoTime()
+        logger.fh    = path.open( fmAppend )
+        logger.start = getMonoTime()
 
-    # Wait for exclusive lock.
+        # Wait for exclusive lock.
-    discard logger.fh.getFileHandle.lockf( F_LOCK, 0 )
+        discard logger.fh.getFileHandle.lockf( F_LOCK, 0 )
-    logger.fh.writeLine "\n", now().utc, " ------------------------------------------------------------------"
+        logger.fh.writeLine "\n", now().utc, " ------------------------------------------------------------------"
+    except IOError as err:
+        echo "Unable to open logfile: $#" % [ err.msg ]
 
 
 proc close*( l: Logger ): void =

src/lib/util.nim

@@ -48,7 +48,70 @@ const
     Display version number.
     """
     EXAMPLECONFIG = """
-sdfdsfsdfsdfsdfdsfdf FIXME: FIXME WJSDFJKSDFKSDF
+#
+# Example Sieb configuration file.
+#
+# This file performs no actions by default, edit to taste.
+#
+
+ 
+## Rules tried before global filtering is applied.
+## You can use this section to deliver before a spamfilter, for example.
+##
+# early_rules:
+#
+#   # Mail that is both from me and to me is questionable....
+#   -
+#     match:
+#       from: mahlon@martini.nu
+#       # Magic "TO" which means To: OR Cc:
+#       TO: mahlon@martini.nu
+#     filter:
+#       - [ reformail, -A, "X-Sieb: I sent mail to myself?" ]
+#     deliver: .suspicious
+#
+#  # No need to spam filter from the FreeBSD mailing list
+#  -
+#    match:
+#      list-id: .*freebsd-questions.*
+#    deliver: .freebsd
+
+
+
+## Global filtering.  All incoming mail that didn't match an "early rule" gets
+## this treatment.
+##
+# filter:
+#   - [ bogofilter, -uep ]
+
+
+
+## Normal rules, after the global filtering.
+## Multiple matches are AND'ed.  Everything is case insensitive.
+##
+## Delivery default is ~/Maildir, any set value is an auto-created maildir under
+## that path.
+##
+# rules:
+#   # More mailing lists, after filters
+#   -
+#     match:
+#       list-id: .*lists.linuxaudio.org.*
+#     deliver: .linuxaudio
+#
+#   # Bye, spam.  (Custom bogofilter header?)
+#   -
+#     match:
+#       x-spamfilter: \+
+#     deliver: .spam
+#
+#   # Just filter the message and deliver to default.
+#   -
+#     match:
+#       x-weird-example: hey, why not
+#       filter:
+#         - [ reformail, -A, "X-Sieb: Well alright then" ]
+
     """
 
 #############################################################