SMTP over Hidden Services with postfix

More and more privacy experts are nowdays calling people to move away from the email service provider giants (gmail, yahoo!, microsoft, etc) and are urging people to set up their own email services, to “decentralize”. This brings up many many other issues though, and one of which is that if only a small group people use a certain email server, even if they use TLS, it’s relatively easy for someone passively monitoring (email) traffic to correlate who (from some server) is communicating with whom (from another server). Even if the connection and the content is protected by TLS and GPG respectively, some people might feel uncomfortable if a third party knew that they are actually communicating (well these people better not use email, but let’s not get carried away).

This post is about sending SMTP traffic between two servers on the Internet over Tor, that is without someone being able to easily see who is sending what to whom. IMHO, it can be helpful in some situations to certain groups of people.

There are numerous posts on the Internet about how you can Torify all the SMTP connections of a postfix server, the problem with this approach is that most exit nodes are blacklisted by RBLs so it’s very probable that the emails sent will either not reach their target or will get marked as spam. Another approach is to create hidden services and make users send emails to each other at their hidden service domains, eg username@a2i4gzo2bmv9as3avx.onion. This is quite uncomfortable for users and it can never get adopted.

There is yet another approach though, the communication could happen over Tor hidden services that real domains are mapped to.

HOWTO
Both sides need to run a Tor client:
aptitude install tor torsocks

The setup is the following, the postmaster on the receiving side sets up a Tor Hidden Service for their SMTP service (receiver). This is easily done in his server (server-A) with the following line in the torrc:
HiddenServicePort 25 25. Let’s call this HiddenService-A (abcdefghijklmn12.onion). He then needs to notify other postmasters of this hidden service.

The postmaster on the sending side (server-B) needs to create 2 things, a torified SMTP service (sender) for postfix and a transport map that will redirect emails sent to domains of server-A to HiddenService-A.

Steps needed to be executed on server-B:
1. Create /usr/lib/postfix/smtp_tor with the following content:

#!/bin/sh

torsocks /usr/lib/postfix/smtp $@

2. Make it executable
chmod +x /usr/lib/postfix/smtp_tor

3. Edit /etc/postfix/master.cf and add a new service entry
smtptor unix - - - - - smtp_tor
For Debian Stretch and/or for postfix 2.11+ this should be:

smtptor      unix  -       -       -       -       -       smtp_tor
  -o smtp_dns_support_level=disabled

4. If you don’t already have a transport map file, create /etc/postfix/transport with content (otherwise just add the following to your transport maps file):

domain-a.net        smtptor:[abcdefghijklmn12.onion]
domain-b.com        smtptor:[bbbcccdddeeeadas.onion]

5. if you don’t already have a transport map file edit /etc/postfix/main.cf and add the following:
transport_maps = hash:/etc/postfix/transport

6. run the following:
postmap /etc/postfix/transport && service postfix reload

7. If you’re running torsocks version 2 you need to set AllowInbound 1 in /etc/tor/torsocks.conf. If you’re using torsocks version 1,you shouldn’t, no changes are necessary.

Conclusion
Well that’s about it, now every email sent from a user of server-B to username@domain-a.net will actually get sent over Tor to server-A on its HiddenService. Since HiddenServices are usually mapped on 127.0.0.1, it will bypass the usual sender restrictions. Depending on the setup of the receiver it might even evade spam detection software, so beware…If both postmasters follow the above steps then all emails sent from users of server-A to users of server-B and vice versa will be sent anonymously over Tor.

There is nothing really new in this post, but I couldn’t find any other posts describing such a setup. Since it requires both sides to actually do something for things to work, I don’t think it can ever be used widely, but it’s still yet another way to take advantage of Tor and Hidden Services.

!Open Relaying
When you setup a tor hidden service to accept connections to your SMTP server, you need to be careful that you aren’t opening your mail server up to be an open relay on the tor network. You need to very carefully inspect your configuration to see if you are allowing 127.0.0.1 connections to relay mail, and if you are, there are a couple ways to stop it.

You can tell if you are allowing 127.0.0.1 to relay mail if you have something like this in your postfix configuration by looking at the smtpd_recipient_restrictions and seeing if you have permit_mynetworks, and your mynetworks variable includes 127.0.0.1/8 (default). The tor hidden service will connect via 127.0.0.1, so if you allow that to send without authentication, you are an open relay on the tor network, and you don’t want that…

Three ways of dealing with this.

1. Remove remove 127.0.0.1 from mynetworks and use port 25/587 as usual.

2. Create a new secondary transport that has a different set of restrictions. Copy the restrictions from main.cf and remove ‘permit_mynetworks’ from them
/etc/postfix/master.cf

2525      inet  n       -       -       -       -       smtpd
   -o smtpd_recipient_restrictions=XXXXXXX
   -o smtpd_sender_restrictions=YYYYYY
   -o smtpd_helo_restrictions=ZZZZ

2587 inet n - - - - smtpd
   -o smtpd_enforce_tls=yes
   -o smtpd_tls_security_level=encrypt
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o smtpd_sender_restrictions=
   -o smtpd_recipient_restrictions=XXXXXXX
   -o smtpd_sender_restrictions=YYYYYY
   -o smtpd_helo_restrictions=XXXXX

Then edit your /etc/tor/torrc

HiddenServiceDir /var/lib/tor/smtp_onion
HiddenServicePort 25 2525
HiddenServicePort 587 2587

3. If your server is not used by other servers to relay email, then you can use the newer postfix variable that was designed for restricting relays smtpd_relay_restrictions (remember NOT to use permit_mynetworks there) to allow emails to be “relayed” by the onion service:
/etc/postfix/main.cf

smtpd_relay_restrictions = permit_sasl_authenticated,
        reject_unauth_destination

smtpd_recipient_restrictions =
        reject_unknown_recipient_domain,
        check_recipient_access hash:$checks_dir/recipient_access,
        permit_sasl_authenticated,
        permit_mynetworks,
        permit

Concerns
Can hidden services scale to support hundreds or thousands of connections e.g. from a mailing list ? who knows…
This type of setup needs the help of big fishes (large independent email providers like Riseup) to protect the small fishes (your own email server). So a new problem arises, bootstrapping and I’m not really sure this problem has any elegant solution. The more servers use this setup though, the more useful it becomes against passive adversaries trying to correlate who communicates with whom.
The above setup works better when there are more than one hidden services running on the receiving side so a passive adversary won’t really know that the incoming traffic is SMTP, eg when you also run a (busy) HTTP server as a hidden service at the same machine.
Hey, where did MX record lookup go ?

Trying it
If anyone wants to try it, you can send me an email using voidgrz25evgseyc.onion as the Hidden SMTP Service (in the transport map).

Links:
http://www.postfix.org/master.5.html
http://www.groovy.net/ww/2012/01/torfixbis
ehloonion/onionmx github repository

*Update 01/02/2015 Added information about !Open Relaying and torsocks version 2 configuration*
*Update 11/10/2016 Updated information about !Open Relaying*
*Update 14/06/2018 Added link to ehloonion/onionmx*

Anonymize headers in postfix

E-mail headers usually leak some information about the person sending the email. Most servers reveal the sender’s originating IP, but sometimes we might not want this behavior. Here’s a simple way to modify your postfix server to remove just the IP of the sender. The original idea is from https://we.riseup.net/debian/mail but with postfix 2.9 version (Debian Wheezy) using the way proposed in the riseup article you will also be anonymizing all intermediate ‘Received: from’ headers and not just the sender’s. The setup proposed by riseup article seems to work fine with postfix 2.7 (Debian Squeeze).

1. Install postfix-pcre if you haven’t already.
# apt-get install postfix-pcre


2.
Create a file /etc/postfix/smtp_header_checks with content:
/^\s*(Received: from)[^\n]*(.*)/ REPLACE $1 [127.0.0.1] (localhost [127.0.0.1])$2


3.
Edit /etc/postfix/master.cf
Find the section about submission and add at the end of it: -o cleanup_service_name=subcleanup
e.g.

submission inet n       -       -       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

submission inet n       -       -       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING
  -o cleanup_service_name=subcleanup

Then at the end of /etc/postfix/master.cf file add the following:

subcleanup unix n       -       -       -       0       cleanup
    -o header_checks=pcre:/etc/postfix/smtp_header_checks

That’s it, reload your postfix and you’re done. When you’ll be sending emails over submission (you do use submission instead of smtp to send your emails, right?) then the first ‘Received’ header will be modified like the following example.
Instead of:

Received: from foo.bar (abcd.efgh.domain.tld [111.222.100.200])
        by mail.domain.tld (Postfix) with ESMTPA id BAB8A1A0224
        for <user@dst.domain2.tld>; Sun, 24 Nov 2013 15:47:50 +0100 (CET)

It will be:

Received: from [127.0.0.1] (localhost [127.0.0.1])
        by mail.domain.tld (Postfix) with ESMTPA id BAB8A1A0224
        for <user@dst.domain2.tld>; Sun, 24 Nov 2013 15:47:50 +0100 (CET)

Extra
If you want to anonymize even more headers, try adding the following to /etc/postfix/smtp_header_checks

/^\s*User-Agent/        IGNORE
/^\s*X-Enigmail/        IGNORE
/^\s*X-Mailer/          IGNORE
/^\s*X-Originating-IP/  IGNORE

Logging
As the riseup article says, be very careful of what is being logged at the server. If you don’t want to log the replacements done by pcre then add something like the following in your rsyslog.conf before any other rule:
:msg, contains, "replace: header Received:" ~

Rate limit outgoing emails from PHP web applications using postfix and policyd

One of the worst things a webmaster or a anyone else that runs some web application can do, is to constantly send “informative newsletters” to people. Most CMS applications make it really easy to send such emails. These are 99% spam, and as such there are many good reasons that you should limit the amount of such outgoing “newsletters” coming out of your email server. Else there’s a good chance you might get added to a blacklist, and you don’t want your legitimate clients to have their emails blocked because of some irresponsible people. I recently had to deploy such a solution to a hosting server that serves multiple (>300) domains. The server already ran postfix, so I had to implement something useful around it.

The problem with postfix is that you can’t really rate-limit the outgoing queue per sender domain/address. There are only generic settings that control the general mail server’s capabilities of sending emails. What I wanted though is to have the ability to restrict specific domains to some specific email message count per day. This is something that a postfix addon named postfix-policyd can do by deferring/greylisting, but still just on the incoming queue. One would think that the problems would be solved by just applying this, but truth is that they don’t. Applying a defer/greylisting policy on the incoming queue is fine while the client on the remote side is another SMTP server that can happily store the deferred email on its queue and retry some minutes/hours later. What happens though if the SMTP client is a PHP application that connects through the mail() function ? There you have no queue and if you defer a message at the SMTP server it will get forever lost, PHP can’t resend it. So the solution would be to apply an intermediate SMTP queue between PHP and the primary SMTP server, that is another local postfix installation that would only serve as a queue that relays emails to the primary.

Using a “simple” diagram sending an email from PHP should follow this path upon a successful installation:

PHP mail() –(sendmail binary)–> intermediate_POSTFIX –(SMTP relay)–> POSTFIX –(smtpd_sender_restrictions)–> POLICYD –(pickup)–> POSTFIX –(SMTP)–> REMOTE SERVER

Here are the steps I took on a Debian Squeeze server to install this little monster.

1. Create a new postfix configuration directory for the new intermediate postfix instance
I named my intermediate postfix config dir as postfix2525, name comes from the port that it will listen on but you can definitely be more creative.

# mkdir /etc/postfix2525
# cp -av /etc/postfix /etc/postfix2525

Remove everything from /etc/postfix2525/main.cf and just add the following lines:

data_directory = /var/lib/postfix2525
queue_directory = /var/spool/postfix2525
relayhost = 127.0.0.1:12525

This defines a new data and queue directory and instructs this postfix to relay all emails through another one that listens on the localhost, the primary one, on port 12525. More about this port later when you will create some special config on the primary postfix.

Remove previous contents of /etc/postfix2525/master.cf and just add these lines:

127.0.0.1:2525      inet  n       -       -       -       2       smtpd
        -o syslog_name=postfix2525
pickup    fifo  n       -       -       60      1       pickup
cleanup   unix  n       -       -       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
#qmgr     fifo  n       -       -       300     1       oqmgr
tlsmgr    unix  -       -       -       1000?   1       tlsmgr
rewrite   unix  -       -       -       -       -       trivial-rewrite
bounce    unix  -       -       -       -       0       bounce
defer     unix  -       -       -       -       0       bounce
trace     unix  -       -       -       -       0       bounce
verify    unix  -       -       -       -       1       verify
flush     unix  n       -       -       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       -       -       -       smtp
# When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay     unix  -       -       -       -       -       smtp
        -o smtp_fallback_relay=
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       -       -       -       showq
error     unix  -       -       -       -       -       error
retry     unix  -       -       -       -       -       error
discard   unix  -       -       -       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       -       -       -       lmtp
anvil     unix  -       -       -       -       1       anvil
scache    unix  -       -       -       -       1       scache

Obviously the most important part here is the first line. It defines that this postfix instance will listen for SMTP connections on localhost, port 2525 and it’s syslog output name will be postfix2525 so that it’s easier to tell apart which SMTP instance spits which errors.

After this is done you need to run the following command that will create all necessary directories with their proper permissions.

# postfix -c /etc/postfix2525/ check

Also make sure you add the following line to the main.cf file of your main postfix installation:
alternate_config_directories = /etc/postfix2525

You will also need a new init script. Since the script by itself is quite big and there are only a few lines that actually differ, I will post my diff here:

--- /etc/init.d/postfix  2011-05-04 21:17:47.000000000 +0200
+++ /etc/init.d/postfix2525  2011-12-19 19:22:09.000000000 +0100
@@ -17,8 +17,10 @@
 # Description:       postfix is a Mail Transport agent
 ### END INIT INFO
 
+CONFDIR=/etc/postfix2525
 PATH=/bin:/usr/bin:/sbin:/usr/sbin
 DAEMON=/usr/sbin/postfix
+DAEMON_OPTIONS="-c /etc/postfix2525"
 NAME=Postfix
 TZ=
 unset TZ
@@ -28,13 +30,13 @@
 
 test -f /etc/default/postfix && . /etc/default/postfix
 
-test -x $DAEMON && test -f /etc/postfix/main.cf || exit 0
+test -x $DAEMON && test -f /etc/postfix2525/main.cf || exit 0
 
 . /lib/lsb/init-functions
 #DISTRO=$(lsb_release -is 2>/dev/null || echo Debian)
 
 running() {
-    queue=$(postconf -h queue_directory 2>/dev/null || echo /var/spool/postfix)
+    queue=$(postconf -c $CONFDIR -h queue_directory 2>/dev/null || echo /var/spool/postfix2525)
     if [ -f ${queue}/pid/master.pid ]; then
   pid=$(sed 's/ //g' ${queue}/pid/master.pid)
   # what directory does the executable live in.  stupid prelink systems.
@@ -66,7 +68,7 @@
       fi
 
       # see if anything is running chrooted.
-      NEED_CHROOT=$(awk '/^[0-9a-z]/ && ($5 ~ "[-yY]") { print "y"; exit}' /etc/postfix/master.cf)
+      NEED_CHROOT=$(awk '/^[0-9a-z]/ && ($5 ~ "[-yY]") { print "y"; exit}' /etc/postfix2525/master.cf)
 
       if [ -n "$NEED_CHROOT" ] && [ -n "$SYNC_CHROOT" ]; then
     # Make sure that the chroot environment is set up correctly.
@@ -111,7 +113,7 @@
     umask $oldumask
       fi
 
-      if start-stop-daemon --start --exec ${DAEMON} -- quiet-quick-start; then
+      if start-stop-daemon --start --exec ${DAEMON} -- ${DAEMON_OPTIONS} quiet-quick-start; then
     log_end_msg 0
       else
     log_end_msg 1
@@ -123,7 +125,7 @@
   RUNNING=$(running)
   log_daemon_msg "Stopping Postfix Mail Transport Agent" postfix
   if [ -n "$RUNNING" ]; then
-      if ${DAEMON} quiet-stop; then
+      if ${DAEMON} ${DAEMON_OPTIONS} quiet-stop; then
     log_end_msg 0
       else
     log_end_msg 1

If everything went well up to now you should be able to start your new postfix instance and check that it is actually running.

# /etc/init.d/postfix2525 start
# netstat -antp | grep 2525
tcp        0      0 127.0.0.1:2525          0.0.0.0:*               LISTEN      6138/master

2. Configure main postfix to accept emails from the intermediate
Edit /etc/postfix/master.cf and add this line at the bottom:

127.0.0.1:12525 inet n - - - - smtpd  -o smtp_fallback_relay= -o smtpd_client_restrictions=  -o smtpd_helo_restrictions=  -o smtpd_recipient_restrictions=permit_mynetworks,reject  -o smtpd_data_restrictions=  -o receive_override_options=no_unknown_recipient_checks

This defines a special port for the main postfix instance that has (or maybe it hasn’t actually) some special restrictions.
Actually you will have to change this line later on upon installing postfix-policyd, but this should be good enough for now, in order for you to do some testing.
Restart postfix

# /etc/init.d/postfix restart
# netstat -antp | grep 2525
tcp        0      0 127.0.0.1:12525         0.0.0.0:*               LISTEN      26799/master    
tcp        0      0 127.0.0.1:2525          0.0.0.0:*               LISTEN      6138/master   

The intermediate postfix listens on 127.0.0.1:2525 and the main one has another special listening port on 127.0.0.1:12525.

3. Test your intermediate postfix instance
You can do this in a gazillion different ways. One of my favorite ways to test SMTP connectivity is through telnet (—> shows data entry):

# telnet localhost 2525
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 server.mydomain.gr ESMTP Postfix
---> EHLO koko.gr
250-server.mydomain.gr
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
---> MAIL FROM: lala@koko.gr
250 2.1.0 Ok
---> RCPT TO: koko@destination.gr
250 2.1.5 Ok
---> DATA
354 End data with <CR><LF>.<CR><LF>
---> THIS IS A TEST
---> .
250 2.0.0 Ok: queued as C41E21C84FF
---> quit

If you were keeping an eye on syslog messages you should have seen some connection messages both from postfix2525 and from postfix. If everything went well your email _should_ have arrived at it’s destination. If this is true then your primary postfix instance now works as a relay for your intermediate queue.

Don’t read the next parts of this post if you haven’t previously managed this step!

4. Install and configure postfix-policyd

# aptitude install postfix-policyd

To run policyd you need to create a database and import policyd SQL schema to it. Your distro has probably already taken care of the previous step, if it hasn’t…do it manually and think about changing distro!
Then edit the config file usually located at /etc/postfix-policyd.conf. The options I chose to play with were the following:
SENDERTHROTTLE=1
SENDER_THROTTLE_SASL=1
SENDER_THROTTLE_HOST=0

Since all emails will be relayed through localhost there’s no point in throttling per host, what is needed is throttling per envelope sender.
You should manually review your desired limits though. I won’t post mine here because everyone has different needs and there’s no sane config for everyone.

Start postfix-policyd
# /etc/init.d/postfix-policyd start

If you get weird startup errors like:
postfix-policyd: fatal: didn't find priority 'LOG_IFOO', exiting
Edit /etc/postfix-policyd.conf, find the following line:
SYSLOG_FACILITY="LOG_MAIL | LOG_INFO"
and change it to (mind the removed spaces):
SYSLOG_FACILITY="LOG_MAIL|LOG_INFO"

5. Configure main postfix instance to use postifix-policyd
Edit /etc/postfix/main.cf and add this:
webclient_restrictions = check_policy_service inet:127.0.0.1:10031

Then edit /etc/postfix/master.cf again and change the line you had previously added to the bottom of the file with this:

127.0.0.1:12525 inet n - - - - smtpd  -o smtp_fallback_relay= -o smtpd_client_restrictions=  -o smtpd_helo_restrictions=  -o smtpd_recipient_restrictions=permit_mynetworks,reject  -o smtpd_data_restrictions=  -o receive_override_options=no_unknown_recipient_checks -o smtpd_sender_restrictions=${webclient_restrictions}

The difference is
-o smtpd_sender_restrictions=${webclient_restrictions}
which practically instructs postfix to use postfix-policyd for emails that arrive on port 12525, which is the port that the intermediate postfix instance uses to relay all emails.

6. Test your intermediate postfix instance again
If everything went well, the main postfix instance should now be able to enforce sender policies. Try sending a new email through the intermediate postfix again, yes using telnet, and you should pickup some new log lines at your syslog:

Dec 19 21:56:40 myserver postfix-policyd: connection from: 127.0.0.1 port: 45635 slots: 0 of 4096 used
Dec 19 21:56:40 myserver postfix-policyd: rcpt=5, greylist=new, host=127.0.0.1 (unknown), from=lala@koko.gr, to=koko@lalala.gr, size=348
Dec 19 21:56:40 myserver postfix/smtpd[9168]: NOQUEUE: reject: RCPT from unknown[127.0.0.1]: 450 4.7.1 : Sender address rejected: Policy Rejection- Please try later.; from= to= proto=ESMTP helo=
Dec 19 21:56:40 myserver postfix/smtp[8970]: C41E21C84FF: to=, relay=127.0.0.1[127.0.0.1]:12525, delay=20, delays=20/0/0.01/0, dsn=4.7.1, status=deferred (host 127.0.0.1[127.0.0.1] said: 450 4.7.1 : Sender address rejected: Policy Rejection- Please try later. (in reply to RCPT TO command))

The above means that greylisting through policyd works.

7. make PHP use your new intermediate postfix instance
PHP on linux by default uses the sendmail binary to send emails via the mail() function. That would use the main postfix instance though, so one needs to edit /etc/php/apache2/php.ini and change the following line:
sendmail_path = "sendmail -C /etc/postfix2525 -t -i"

The -C directive instructs sendmail to use the alternate config dir, so that emails will be sent to the new intermediate postfix instance and then to the main one, passing through policyd of course.

To check the queue size of the intermediate postfix:
# postqueue -p -c /etc/postfix2525/

If any PHP applications that are hosted have explicit SMTP server/port directives, then be sure to notify your clients/developers that they _MUST_ use localhost:2525 to send their emails to and not the default localhost:25. This is one of the shortcomings of the above method, if someone manually sets up his application to use the default localhost:25 his emails will get right through. But being a good sysadmin, you should monitor such behavior and punish those users accordingly!

That’s about it…with the above configuration and some tweaking to the thresholds you have very good chances of avoiding getting blacklisted because someone decided to send a few thousand spams emails. And most importantly, your normal mail service will continue to work flawlessly, no matter how big the queue of the intermediate mail server is.

Enjoy!

Reference for policyd: http://policyd.sourceforge.net/readme.html