SMTP, TLS and STARTTLS (server side), behavior and setup

b4u

Golden Member
Nov 8, 2002
1,380
2
81
Hi,

Hope I can find some help in here, in trying to understand how SMTP and TLS security should be setup.

My intention is to setup a simple email server for personal use. I'll setup two machines, on different places, on primary MX and a secondary (backup) one.

So for now, a simple test, with a single MX record and a small test server:
1. Setup a mail server, with SMTP listening on port 25;
2. Created a self signed certificate, and keystore'd it with a password;
3. Setup SMTP service to "EnableTLS", and "RequireTLS" settings;
4. Setup my router to deliver port 25 TCP traffic to the test server;

My intentions are to have a personal mail server, where the emails are delivered to 1-2 mailboxes, using TLS for security. That is, I want to be sure the emails are delivered securely on my server (although they might be floating around the web, with no security restrictions).

For the server software I used "Mireka" http://code.google.com/p/mireka/, an open source server made in java. I enjoyed the simple configuration, and the simplicity. Being a java developer myself, I see the potential to evolve the software to my needs, present and future ones.



So in my tests, I used telnet, openssl and an external "outlook.com" (hotmail-based?) account, sending mails as described in the following scenarios:

# Scenario-A
Without "RequireTLS" setting from server, sent an email to a test mailbox, with SUCCESS (Expected).

Using telnet to port 25 displayed "250-STARTTLS" after EHLO, but I didn't execute that command, so I assume no encryption was in place for receiving mail.


# Scenario-B
With "RequireTLS" setting on server, sent an email to a test mailbox, with FAILURE (Expected).

Using the same approach telnet to port 25 displayed "250-STARTTLS" after EHLO.

After sending "MAIL TO" I received the "530 Must issue a STARTTLS command first", which means the server requests TLS to proceed, making sure security encryption will be in place for message delivery.

# Scenario-C
With "RequireTLS" setting on server, sent an email to a test mailbox, with SUCCESS (Expected).

Using "openssl s_client -starttls smtp -connect ..." will connect, issue a STARTTLS and complete the proper handshake.

Issuing the required SMTP commands, I was able to send a message, and so assuming security encryption was in place, as I want it.


Now for the confusing part ...

# Scenario-D
Sending from an external email service will FAIL (Not Expected)

I went to an "outlook.com" (hotmail-based?) account, and sent emails from there, with "RequireTLS" setting on server.

The mails are returned, and on the server side I can check on the logs that STARTTLS is not called from the client side (some hotmail sender), so a "530 Must issue a STARTTLS command first" is returned and the connection refused.

If I remove "RequireTLS" setting, the emails are delivered, but no STARTTLS is called, hence no security encryption is applied.



So I'm a bit confused as how this should be setup to work:

1. Shouldn't STARTTLS be issued when it is available, or at least when a "530" is returned, requesting for a STARTTLS?

2. The MX record is defined, and I have port 25 configured for external access. This port is called as it is the default port for MX entries, but should I set port 465 instead (deactivating 25) so that external connections will know and force SSL/TLS?


Any help appreciated, any tips are welcomed.

Thanks.
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
1) STARTTLS is optional. The sending server has to support it, have it configured etc. Free services generally won't because the cert chain isn't free and the SSL encryption isn't trivial when applied to millions of emails a day.
2) Port 25 is MTA to MTA communications. 465 is for client use. In email, you basically always use 25. If you don't you basically don't exist on the internet.

In addition if your cert is not trusted ie self signed etc it might not be used. Even if starttls is allowed / enabled.
 
Last edited:

b4u

Golden Member
Nov 8, 2002
1,380
2
81
1) STARTTLS is optional. The sending server has to support it, have it configured etc. Free services generally won't because the cert chain isn't free and the SSL encryption isn't trivial when applied to millions of emails a day.
2) Port 25 is MTA to MTA communications. 465 is for client use. In email, you basically always use 25. If you don't you basically don't exist on the internet.

In addition if your cert is not trusted ie self signed etc it might not be used. Even if starttls is allowed / enabled.


Thanks for your reply.

During my tests, I've successfully sent an email using javamail API, using TLS, but yes, I had to add the server certificate as a trusted one.

I was thinking of an alternative, to update the server side code, so that one a client connects to SMTP server, I attempt to create an SSL Socket, forcing the handshake, but I believe it's a long shoot in the dark, and most probably the client will not accept the certificate, failing to connect.

If my certificate was a properly CA signed one, from a trusted source, would that assure it's acceptance in SMTP secure connection?

So basically from what I see, basically there is no way I can assure a secure email transfer unless I encrypt the message contents (which is always the best approach).

I'm safe to assume then, that the most common usage of TLS for email exchange is on a SMTP relay/sender server, or when connecting known SMTP relay servers between each other.


Thanks.
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
Thanks for your reply.

During my tests, I've successfully sent an email using javamail API, using TLS, but yes, I had to add the server certificate as a trusted one.

I was thinking of an alternative, to update the server side code, so that one a client connects to SMTP server, I attempt to create an SSL Socket, forcing the handshake, but I believe it's a long shoot in the dark, and most probably the client will not accept the certificate, failing to connect.

If the sending MTA doesn't support STARTTLS, it will simply drop the connection if you force it.

If my certificate was a properly CA signed one, from a trusted source, would that assure it's acceptance in SMTP secure connection?

Nothing is guaranteed but it would greatly increase the odds. Most servers will verify the certificate chain, date, server name etc.

So basically from what I see, basically there is no way I can assure a secure email transfer unless I encrypt the message contents (which is always the best approach).

I'm safe to assume then, that the most common usage of TLS for email exchange is on a SMTP relay/sender server, or when connecting known SMTP relay servers between each other.


Thanks.

You can force TLS, the main thing is if the sending server or relay doesn't support it, you won't get the message. This will make sure everything that is sent is encrypted but there is no way to guarantee delivery in this config. Clients can be forced to the encrypted ports to force encryption there.

Also be aware that the message is decrypted every step of the way to set up the next hop. If you want the message encrypted, you need to encrypt that and not the transmissions only.