June 18, 2025 | Scott

How to Set Up a Mail Server on a cPanel VPS: A Step-by-Step Guide for Administrators

Setting up a mail server on a cPanel VPS isn’t a casual admin task. It requires precision, awareness of DNS and authentication standards, and a solid understanding of how mail flow works. The reward? Full control over your email infrastructure, improved security, and better deliverability. In this guide, I’ll walk you through configuring your VPS to handle mail properly using cPanel/WHM.

1. Prerequisites

Before doing anything, ensure your VPS meets the following:

Root Access: Full administrative privileges are required.
cPanel/WHM Installed: Mail server configuration is tied to WHM’s services.
Valid Hostname: Must be an FQDN like server.example.com.
Reverse DNS: Set via your VPS provider. Without it, most major mail providers will flag your emails.
Dedicated IP (recommended): Shared IPs hurt deliverability.

Once these are ready, log into WHM as root.

2. Configure Your Hostname and RDNS

Hostname

Navigate to:
WHM → Networking Setup → Change Hostname

Use a valid, unique FQDN. Do not use your domain (example.com) directly—use mail.example.com or server.example.com.

Reverse DNS (PTR)

Configure this via your VPS host’s dashboard or support ticket. Match it to your hostname. For example, if your hostname is mail.example.com, the PTR record should resolve your IP to mail.example.com.

3. Add the Domain to cPanel

If not already done, add the domain:

WHM → Create a New Account
Fill in the domain, username, password, and allocate resources.
Click Create.

This automatically sets up default email services for the domain.

4. Configure MX Records

Head to:
WHM → DNS Functions → Edit DNS Zone

Make sure the MX record for your domain points to your mail server hostname:

example.com. 14400 IN MX 10 mail.example.com.

Also, ensure there’s an A record:

mail 14400 IN A [your server IP]

5. Enable SPF, DKIM, and DMARC

Without these, expect to land in spam folders—or be outright blocked.

SPF

SPF tells other mail servers which IPs are allowed to send mail for your domain.

Go to WHM → Email → Email Deliverability
Click “Manage” next to your domain.
Ensure SPF is installed and says something like:

v=spf1 +a +mx +ip4:[your_server_ip] ~all

DKIM

DKIM signs your outgoing messages. It’s also found in Email Deliverability. WHM will generate the key and publish the DNS record.

Ensure DKIM is active and properly propagated in DNS.

DMARC (Manual Step)

Create a DMARC TXT record via cPanel → Zone Editor or WHM → Edit DNS Zone:

_dmarc.example.com. 14400 IN TXT "v=DMARC1; p=none; rua=mailto:postmaster@example.com"

Change p=none to p=quarantine or p=reject once confident in SPF/DKIM.

6. Configure Mail Routing

Each domain on your server can either use local mail routing (handled by the VPS) or remote (e.g., using Google Workspace). For your mail server:

Go to WHM → Edit DNS Zone → [your domain]
Scroll to Email Routing.
Choose Local Mail Exchanger.

This tells Exim (the mail transport agent) to accept mail for your domain.

7. Secure the Mail Server

Security is non-negotiable. Here’s what to configure:

Enable TLS

Exim and Dovecot must support encrypted connections. WHM handles this:

Go to WHM → Service Configuration → Mailserver Configuration
Ensure TLS support is enabled.

Set up AutoSSL

WHM includes AutoSSL via cPanel:

WHM → SSL/TLS → Manage AutoSSL
Use cPanel (powered by Sectigo) or Let’s Encrypt.
Enable for all users.

AutoSSL ensures mail clients (and webmail) connect securely.

8. Configure Mail Clients and Ports

Incoming Mail (IMAP/POP3)

IMAP: mail.example.com, Port 993 (SSL)
POP3: mail.example.com, Port 995 (SSL)

Outgoing Mail (SMTP)

SMTP: mail.example.com, Port 465 (SSL) or 587 (STARTTLS)

Use full email address and password for authentication.

If you’re using Outlook, Apple Mail, or Thunderbird, most can autodetect these settings from your DNS and SSL.

9. Spam and Virus Filtering

WHM includes SpamAssassin and ClamAV.

SpamAssassin

WHM → Service Configuration → Exim Configuration Manager → SpamAssassin Options
Enable and adjust the spam score threshold (default is 5).

ClamAV

Install via WHM → Plugins → Manage Plugins → ClamAV
Once installed, it integrates with mail scanning.

Both tools can reduce spam and protect your server from mail-borne malware.

10. Test Deliverability

Before sending real mail, test everything:

Use Mail-Tester.com to analyze SPF, DKIM, PTR, and spam score.
Send test emails to Gmail, Outlook, and ProtonMail.
Check if mail lands in Inbox or Spam.
Review email headers to trace authentication results.

If emails go to spam, check:

SPF/DKIM records
RDNS match
Mail content (avoid spammy phrases)

11. Monitor and Maintain

Log Files

Exim logs: /var/log/exim_mainlog
Dovecot logs: /var/log/maillog or /var/log/messages

Use tail -f to monitor:

tail -f /var/log/exim_mainlog

Email Queue

Check stuck mail:

exim -bp

Force send:

exim -qf

Remove from queue:

exim -Mrm <message-id>

Final Thoughts

Running your own mail server on a cPanel VPS puts you in full control, but it demands responsibility. Misconfiguration can lead to blacklisting or undelivered mail. When done right, it provides a professional, secure, and private way to handle email for your domain.

Use monitoring tools, stay updated with mail server best practices, and audit your DNS records regularly. The difference between reliable mail and a blacklisted server is often one overlooked setting.

Let the big providers worry about scale. Your job is to stay lean, fast, and secure.

Category:Email Hosting

VirtuSky