Setting up an Amazon Ec2 Mail Server
Getting Started
This guide will help you implement a SMTP server with Postfix, and a IMAP server with Cyrus-Imap. Its focus is doing this on a vanilla Ec2 instance with an Amazon AMI distro installed. This setup will support multiple email address from various domains. My goal with this guide is not just to show you how I setup a full featured mail server, but the tools and techniques I use to diganiose problems when you encounter them.
- Server: Amazon AMI t2.micro instance
- john@johnminchuk.com
- john@everyonealive.com
Protocols and Terminology
A quick primer on the different protocols you'll encounter when working with linux mail distribution.
- Sending Mail
- SMTP: sends mail, postfix and sendmail both implement SMTP servers
- Downloading Mail
- POP3: downloading messages off a server, emails removed off server and onto your reading client
- IMAP: reading messages off a server, not neccessarily deleted after being read
- Cyrus-imap implements a IMAP server, Dovecot implements a IMAP and POP server
- Clients
- mail: linux command, can read mail sent to user accounts on a linux server
- mutt and pine: linux programs that make reading email on a linux server easier
- Outlook, Outlook Express, and Windows Live Mail, Thunderbird, Osx Mail: clients running on personal computers that send mail via SMTP, and download it via IMAP or POP3
Sendmail vs Posfix
What are these programs and whats the difference? They're both MTA (mail transfer agents) and basically they control how mail is sent and recieved on your machine. They can tie into different authentication systems and different delivery systems. Sendmail has been around since the 80s and has signs of it's age. Postfix was released in 1998 and contains many improvements over sendmail.
sendmail: old, ugly configuration, common on most linux distros
postfix: newer, more "moving parts", better documentation and resources (in my opinion)
Starting the Ec2 Instance
Let's login to the aws console and start a new ec2 instace.
If you're a t2 micro instance, you'll be launching into a VPC. To get an elastic ip address assigned to this machine visit the VPC console and allocate and associate with your new instance.
Don't forget to assign it to your newly launched Ec2 instance.
* If you launch a t2.micro instance, you'll be in a VPC, you can only get elastic ips from there. Regular Ec2 instances get their elastic ips from the Ec2 page.
At this point you'll be assigned a new public dns based on your elasic ip. you will need to assign this as the hostname of your machine and restart.
sudo vi /etc/sysconfig/network
HOSTNAME=ec2-52-4-179-143.compute-1.amazonaws.com
sudo reboot
Now let's get the software we need installed on the server.
yum install -y telnet mailx mlocate
yum install -y postfix cyrus-imapd cyrus-sasl-md5 cyrus-sasl-plain
[root@ec2-52-4-179-143 ~]# updatedb
We'll need telnet for debugging, mailx for the local mail command to test with, and I like mlocate because it indexs your machine making config files easy to find (this command is optional, run updatedb to get it started).
Most importantly we installed postfix, the imapd server, and some authentication methods that it will use.
Let's take a look at what's running on our server now.
[root@ip-172-30-0-197 ~]# service --status-all acpid is stopped atd (pid 2155) is running... auditd (pid 1915) is running... cfn-hup is stopped Checking for service cloud-init:Checking for service cloud-init:Checking for service cloud-init:Checking for service cloud-init:crond (pid 2145) is running... cyrus-imapd is stopped ip6tables: Firewall is not running. iptables: Firewall is not running. irqbalance is stopped lvmetad is stopped mdmonitor is stopped messagebus (pid 1961) is running... netconsole module not loaded Configured devices: lo eth0 Currently active devices: lo eth0 ntpd (pid 2115) is running... master is stopped Process accounting is disabled. rdisc is stopped rngd (pid 1949) is running... rsyslogd (pid 1933) is running... saslauthd is stopped sendmail (pid 2130) is running... sm-client (pid 2137) is running... openssh-daemon (pid 2098) is running...
We won't be using sendmail let's stop it and stop it from starting with the os. Let's also start cyrus-imapd and set it to start with the os.
[root@ip-172-30-0-197 ~]# service sendmail stop Shutting down sm-client: [ OK ] Shutting down sendmail: [ OK ]
[root@ip-172-30-0-197 ~]# chkconfig sendmail off
[root@ip-172-30-0-197 ~]# service postfix start Starting postfix: [ OK ] [root@ip-172-30-0-197 ~]# chkconfig postfix on
[root@ip-172-30-0-197 ~]# service cyrus-imapd start Importing cyrus-imapd databases: [ OK ] Starting cyrus-imapd: [ OK ] [root@ip-172-30-0-197 ~]# chkconfig cyrus-imapd on
Configure SMTP Authenticaiton
SMTP servers listen on port 25 and respond to mail requets. This could be from mail clients like outlook, or other server using running smtp servers. We need a way to protect this port so spammers cant use this to send unsolicted junk email. Postfix uses smtpd.conf when it starts so we will configure authentication starting there.
[ec2-user@ec2-52-4-146-220 postfix]$ vi /etc/sasl2/smtpd.conf
pwcheck_method: auxprop mech_list: plain login cram-md5 digest-md5
[root@ec2-52-4-179-143 ~]# vi /etc/postfix/main.cf smtpd_sasl_auth_enable = yes
Auxprop says we will be using something other than the local log in system for user account, (as in you will not be creating linux user accounts with useradd to add email accounts, it will be managed separatley). The mech_list prop lists what we will accept for user to login with. plain, login (legacy), cram-md5 and digest-md5 (an ecrypted handshake method). When postfix starts your smtp server you should see this listed. Matter of fact lets try that.
Here we telnet into postfix's smpt server on 25 and manually issue commands to connect to it.
[root@ec2-52-4-179-143 postfix]# service postfix restart Shutting down postfix: [ OK ] Starting postfix: [ OK ] [root@ec2-52-4-179-143 postfix]# telnet localhost 25 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 ec2-52-4-179-143.compute-1.amazonaws.com ESMTP Postfix ehlo dummy.com 250-ec2-52-4-179-143.compute-1.amazonaws.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-AUTH PLAIN CRAM-MD5 DIGEST-MD5 LOGIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN quit Connection closed by foreign host.
Now let's add a user account to log in with...
[root@ec2-52-4-179-143 postfix]# saslpasswd2 john@everyonealive.com Password: Again (for verification):
This will create a new sasl database located here
[root@ec2-52-4-179-143 etc]# ls -l /etc/sasldb2 -rw-r----- 1 root root 12288 Mar 31 01:29 /etc/sasldb2
You can see it has the user account we just created in it
[root@ec2-52-4-179-143 etc]# sasldblistusers2 -f /etc/sasldb2 john@everyonealive.com: userPassword
Let's try logging in with this manually over port 25
[root@ec2-52-4-179-143 etc]# telnet localhost 25 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 ec2-52-4-179-143.compute-1.amazonaws.com ESMTP Postfix ehlo dummy.com 250-ec2-52-4-179-143.compute-1.amazonaws.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-AUTH PLAIN CRAM-MD5 DIGEST-MD5 LOGIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN auth login 334 VXNlcm5hbWU6 am9obkBlYzItNTItNC0xNzktMTQzLmNvbXB1dGUtMS5hbWF6b25hd3MuY29t 334 UGFzc3dvcmQ6 Mzcasdf2a3Q= 535 5.7.8 Error: authentication failed: authentication failure
Wait! It failed! Ok, let's go check out the problem...
[root@ec2-52-4-179-143 ~]# vi /var/log/maillog
Mar 31 01:37:29 ec2-52-4-179-143 postfix/smtpd[3243]: warning: SASL authentication problem: unable to open Berkeley db /etc/sasldb2: Permission denied Mar 31 01:37:29 ec2-52-4-179-143 postfix/smtpd[3243]: warning: localhost[127.0.0.1]: SASL login authentication failed: authentication failure M
Looks like the cyrus user doesn't have access to read this file. Lets fix that. We can see the postfix user belongs to the mail group. Lets update /etc/sasldb to belong to this group.
[root@ec2-52-4-179-143 etc]# cat /etc/group | grep mail mail:x:12:mail,postfix [root@ec2-52-4-179-143 etc]# chgrp mail /etc/sasldb2
Ok, now let's try logging in again.
[root@ec2-52-4-179-143 etc]# telnet localhost 25 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 ec2-52-4-179-143.compute-1.amazonaws.com ESMTP Postfix ehlo dummy.com 250-ec2-52-4-179-143.compute-1.amazonaws.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-AUTH PLAIN CRAM-MD5 DIGEST-MD5 LOGIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN auth login 334 VXNlcm5hbWU6 am9obkBlYzItNTItNC0xNzktMTQzLmNvbXB1dGUtMS5hbWF6b25hd3MuY29t 334 UGFzc3dvcmQ6 Mzcasdf2a3Q= 535 5.7.8 Error: authentication failed: authentication failure auth login 334 VXNlcm5hbWU6 am9obkBlYzItNTItNC0xNzktMTQzLmNvbXB1dGUtMS5hbWF6b25hd3MuY29t 334 UGFzc3dvcmQ6 Mzcasdf2a3Q= 235 2.7.0 Authentication successful
Cool, looks like we're authenticating. Now let's get it working across the internet. Open up port 25 to your Ec2 instance. Matter of fact, while were here, let's open up the IMAP port as well.
Now try telneting from your local machine
$ telnet ec2-52-4-179-143.compute-1.amazonaws.com 25 Trying 52.4.179.143... telnet: connect to address 52.4.179.143: Connection refused
We'll that didn't work. Looks like we need to configure postfix to accept connections from all addresses, not just the local address like we've been testing with. You can see this using netstat.
[root@ec2-52-4-179-143 etc]# netstat -anp | grep :25 tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 3225/master
We can see "master" with process id of 3225 is listening on 127.0.0.1 with port 25. That means port 25 will only accept connections from that address.
[root@ec2-52-4-179-143 etc]# vi /etc/postfix/main.cf
smtpd_sasl_auth_enable = yes broken_sasl_auth_clients = yes smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination ...
inet_interfaces = all #inet_interfaces = $myhostname #inet_interfaces = $myhostname, localhost #inet_interfaces = localhost
[root@ec2-52-4-179-143 etc]# service postfix restart Shutting down postfix: [ OK ] Starting postfix: [ OK ]
Now on your local machine...
$ telnet ec2-52-4-179-143.compute-1.amazonaws.com 25 Trying 52.4.179.143... Connected to ec2-52-4-179-143.compute-1.amazonaws.com. Escape character is '^]'. 220 ec2-52-4-179-143.compute-1.amazonaws.com ESMTP Postfix
Setting up IMAP to read emails
Ok, now were at a point where we can try to read messages sent to the email accounts we setup.
[root@ec2-52-4-179-143 postfix]# vi /etc/postfix/main.cf
virtual_transport = lmtp:unix:/var/lib/imap/socket/lmtp virtual_mailbox_domains = johnminchuk.com everyonealive.com virtual_mailbox_maps = hash:/etc/postfix/johnminchuk.com hash:/etc/postfix/everyonealive.com virtual_alias_maps = hash:/etc/postfix/virtual
Here were saying use protocol lmtp to send messages on this socket (picked up by cyrus-imap). We list the virtual domains we will be accepting mail for, then where we can find hash lists of users we will be accepting mail for (we don't want to accept email for anybody, just whos specified in these mailbox maps). Lastly we will create a virtual map of user accounts pointing to other user account (eg: john@johnminchuk.com could go to john@everyonealive.com as well).
For each of the domains listed above create the map file then use the command below to create the database it will be read from.
[root@ec2-52-4-179-143 postfix]# vi /etc/postfix/johnminchuk.com john@johnminchuk.com john m
[root@ec2-52-4-179-143 postfix]# postmap /etc/postfix/johnminchuk.com
[root@ec2-52-4-179-143 postfix]# ls -l /etc/postfix/johnminchuk.com.db -rw-r--r-- 1 root root 12288 Mar 31 03:24 /etc/postfix/johnminchuk.com.db
Above you can see we created a text file with an email that johnminchuk.com will accept "john@johnminchuk.com" then whitespace, then a name. Afterwards we generate a .db file by using the postmap command.
Now let's configure imap to accept connections. Compare these settings with yours.
[root@ec2-52-4-146-220 postfix]# vi /etc/imapd.conf
configdirectory: /var/lib/imap partition-default: /var/spool/imap admins: cyrus cyrusadm sievedir: /var/lib/imap/sieve sendmail: /usr/sbin/sendmail hashimapspool: true sasl_pwcheck_method: auxprop sasl_mech_list: LOGIN PLAIN CRAM-MD5 DIGEST-MD5 tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt virtdomains: yes defaultdomain: ec2-52-4-179-143.compute-1.amazonaws.com unixhierarchysep: yes allowanonymouslogin: no allowplaintext: yes # uncomment this if you're operating in a DSCP environment (RFC-4594) # qosmarking: af13
Now let's create an admin account as listed in the imapd.conf file, then log in with it.
[root@ec2-52-4-146-220 postfix]# saslpasswd2 cyrusadm Password: Again (for verification): [root@ec2-52-4-179-143 /]# sasldblistusers2 -f /etc/sasldb2
cyrusadm@ec2-52-4-179-143.compute-1.amazonaws.com: userPassword
Now that our sasldb2 database has an admin user, lets log in with that account and create the mail boxes for our two emails.
[root@ec2-52-4-179-143 postfix]# service cyrus-imapd restart Shutting down cyrus-imapd: [ OK ] Exporting cyrus-imapd databases: [ OK ] Importing cyrus-imapd databases: [ OK ] Starting cyrus-imapd: [ OK ] [root@ec2-52-4-179-143 postfix]# cyradm --user=cyrusadm localhost Password: localhost> createmailbox user/john@johnminchuk.com localhost> createmailbox user/john@everyonealive.com
Note that in the createmailbox command above, documentation says "createmailbox user.john@everyonealive.com", however I was unable to get mail delivered to that account. The format user/<email> seems to automatically get email depositied into it, and also shows up in mail clients, with less configuration. See more options for this command here, such as listing all emails and deleting existing ones if neccessary.
If you get a message requesting "IMAP Password", something went wrong creating or authenticating with your new user admin. Make sure the name of the account is correct, double check it exists in /etc/sasldb2, and connect via localhost, I had issues using the hostname of the machine. Check /var/log/maillog for clues.
[root@ec2-52-4-179-143 postfix]# cyradm --user=cyrusadm localhost
Password:
IMAP Password:
Ok let's not forget to create the accounts that will be allowed login to the sytem
[root@ec2-52-4-179-143 postfix]# saslpasswd2 john@everyonealive.com Password: Again (for verification): [root@ec2-52-4-179-143 postfix]# saslpasswd2 john@johnminchuk.com Password: Again (for verification): [root@ec2-52-4-179-143 postfix]# sasldblistusers2 -f /etc/sasldb2 john@everyonealive.com: userPassword john@johnminchuk.com: userPassword cyrusadm@ec2-52-4-179-143.compute-1.amazonaws.com: userPassword
Ok, we can see the new accounts in the auth db. Let's test connectivity.
[root@ec2-52-4-179-143 postfix]# imtest -m login -a john@everyonealive.com localhost S: * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS AUTH=PLAIN AUTH=CRAM-MD5 AUTH=DIGEST-MD5 AUTH=LOGIN SASL-IR COMPRESS=DEFLATE] ec2-52-4-179-143.compute-1.amazonaws.com Cyrus IMAP v2.3.16-Fedora-RPM-2.3.16-6.9.amzn1 server ready Please enter your password: C: L01 LOGIN john@everyonealive.com S: + go ahead C: <omitted> S: L01 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH] User logged in Authenticated. Security strength factor: 0 . logout * BYE LOGOUT received . OK Completed Connection closed.
Looks like we were able to authenticate. Type ". logout" to close the connection. See the Cyrus documentation for more details.
At this point it also wouldn't hurt to verify that you can connect to the IMAP server on port 143 from outside the network. Don't forget to open port 143 to your machine just like we did with port 25.
Setting DNS to point to your new mail server
Let's get DNS pointing correctly at our server. I'll be using Amazon's scalable DNS hosting called Route 53.
Testing with Email Clients and Mobile Devices
Almost there, time to test! Let's download and install Windows Live Mail (a replacement for Outlook Express), which is now part of the Windows Essentials addon package. This example is using Windows 7.
<screen shots>
Note: I had trouble with this behind a corporate firewall. You may want to try it from home.
Adding these accounts to a mobile application should be no different than a desktop client. Same configuration, however, during my tests on an iPhone 5, the settings kept defaulting to an SSL connection for the SMTP server. This caused issues trying to send mail from the phone. You may need to let the phone try to connect, then go an correct it's settings after it has been setup.
Adding a new domain and email
Here's a quick list of steps to get a new domain and email address up and running. We're adding admin@victorianrecords.com
[root@ec2-52-4-179-143 domain]# vi /etc/postfix/main.cf
virtual_mailbox_domains =
johnminchuk.com
everyonealive.com
victorianrecords.com
virtual_mailbox_maps =
hash:/etc/postfix/johnminchuk.com
hash:/etc/postfix/everyonealive.com
hash:/etc/postfix/victorianrecords.com
[root@ec2-52-4-179-143 postfix]# vi /etc/postfix/victorianrecords.com
admin@victorianrecords.com admin
[root@ec2-52-4-179-143 postfix]# postmap /etc/postfix/victorianrecords.com
[root@ec2-52-4-179-143 postfix]# cyradm --user=cyrusadm localhost
Password:
localhost> createmailbox user/admin@victorianrecords.com
localhost> quit
[root@ec2-52-4-179-143 postfix]# ls /var/spool/imap/domain/v/victorianrecords.com/a/user/admin/
[root@ec2-52-4-179-143 postfix]# saslpasswd2 admin@victorianrecords.com
Password:
Again (for verification):
[root@ec2-52-4-179-143 postfix]# service postfix restart
Shutting down postfix: [ OK ]
Starting postfix: [ OK ]
[root@ec2-52-4-179-143 postfix]# service cyrus-imapd restart
Shutting down cyrus-imapd: [ OK ]
Exporting cyrus-imapd databases: [ OK ]
Importing cyrus-imapd databases: [ OK ]
Starting cyrus-imapd: [ OK ]
And don't forget to update your mx records!!!
Port forwarding updates
I had a few issues dealing with Amazon support and allowing connections to my mail server. Amazon has a form you can fill out to request the ability to send email outbound from your server. However, after filling out that form, it seems they took it upon themselfs to block connections inbound on 25. I sent several emails about this issue, but we seemed to be confusing each other. Utimatly my request was sent to senior engineers and they informed me that they recommend using port 587. So I recommend...
- Putting in a request to send mail outbound from your ec2 instance with this form (more on the reverse dns fields later)
- Setting postfix to accept connections on port 587, this will allow your clients (like your phone) to compose a message, then connect to port 587 where it will be forwarded to the rest of the world.
vi /etc/postfix/master.cf //uncomment #submission inet n - n - - smtpd service postfix restart
Now I recommend testing if you can connect to the machine with telnet on port 587. Don't forget to set your clients SMTP to port 587 instead of 25.
Reverse DNS Request
Some email servers are more strict about who they accept email from. Currently amazon sets up your instance with a DNS record pointing your hostname to your elastic ip, something like...
ec2-52-4-179-143.compute-1.amazonaws.com -> 52.4.179.143
Some email servers accepting your message will want to check that your IP address comes from a valid domain, you can do this with a reverse DNS entry. On the aws email request form, make sure to specify the ip of your machine as the elastic ip and the reverse DNS as the hostname of your machine.
Todo
- Using the mail command to read and send email
- Using mutt
- Sending an email with the mail command inline