Email Server Setup with Mailcow: Complete Self-Hosted Guide

Running your own email server gives you complete control over your communications. No third-party reading your emails, no monthly fees per mailbox, and no vendor lock-in.

I've been running Mailcow for several client projects and my own domains. It's the best solution for self-hosted email in 2025. This guide walks you through complete email server setup - from bare server to production-ready mail system, based on what I've learned from real deployments.

Why Self-Host Your Email Server?

Before diving into email server setup, let's understand why you might want to run your own mail server:

Privacy: Your emails stay on your server. No Google, Microsoft, or other providers scanning your messages.

Cost savings: Pay for server only. No per-user fees. A $20/month VPS can handle hundreds of mailboxes.

Custom domains: Unlimited domains and aliases at no extra cost.

Full control: Your rules, your spam filtering, your backup strategy.

Learning: Deep understanding of how email actually works.

The trade-offs: You're responsible for security, updates, and deliverability. I found that once you get past the initial setup, the maintenance is minimal - this guide helps you handle it.

Why Mailcow for Email Server Setup?

Mailcow is a dockerized email server suite that includes:

• Postfix - SMTP server (sending/receiving)
• Dovecot - IMAP/POP3 server (mailbox access)
• SOGo - Webmail and calendar
• Rspamd - Spam filtering
• ClamAV - Antivirus
• Unbound - DNS resolver
• Web UI - Easy administration

All components work together out of the box. Docker makes deployment and updates simple.

Alternatives considered:

• Mail-in-a-Box: Simpler but less flexible
• iRedMail: Good but more complex setup
• Zimbra: Enterprise-focused, heavy
• Manual setup: Weeks of configuration

Mailcow hits the sweet spot of features and manageability for email server setup.

Prerequisites for Email Server Setup

Server Requirements

Minimum specs:

• 2 CPU cores
• 4GB RAM (6GB recommended)
• 20GB SSD storage
• Clean IP address (not blacklisted)
• IPv4 address (IPv6 optional but recommended)

Operating system:

• Ubuntu 22.04 LTS (recommended)
• Debian 11 or 12
• CentOS Stream 9

Recommended providers:

• Hetzner - Best value, good IPs (my go-to choice)
• OVH - Budget-friendly
• DigitalOcean - Easy but pricier
• Vultr - Good IP reputation

Avoid: AWS, Azure, GCP for email - their IPs are often blacklisted for spam. I learned this the hard way when a client's AWS instance couldn't deliver to Gmail for weeks.

Domain Requirements

You need a domain with full DNS control:

• Ability to create A, MX, TXT, CNAME records
• Access to reverse DNS (PTR) - usually via hosting provider
• No existing MX records conflicting

Clean IP Check

Before email server setup, verify your IP isn't blacklisted:

# Check your server IP
curl ifconfig.me

# Then check at:
# https://mxtoolbox.com/blacklists.aspx
# https://www.spamhaus.org/lookup/

If blacklisted, get a new server or request IP change from provider.

Step 1: Prepare the Server

Initial Server Setup

# Update system
apt update && apt upgrade -y

# Set hostname to your mail domain
hostnamectl set-hostname mail.yourdomain.com

# Update /etc/hosts
echo "YOUR_SERVER_IP mail.yourdomain.com mail" >> /etc/hosts

# Verify
hostname -f
# Should output: mail.yourdomain.com

Install Docker

Mailcow runs entirely in Docker containers:

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Add user to docker group
sudo usermod -aG docker $USER

# Install Docker Compose plugin
sudo apt install docker-compose-plugin -y

# Verify
docker --version
docker compose version

Configure Firewall

Open required ports for email server setup:

# Install UFW if needed
apt install ufw -y

# Configure rules
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp    # SSH
ufw allow 80/tcp    # HTTP (ACME/webmail)
ufw allow 443/tcp   # HTTPS (webmail)
ufw allow 25/tcp    # SMTP
ufw allow 465/tcp   # SMTPS
ufw allow 587/tcp   # Submission
ufw allow 993/tcp   # IMAPS
ufw allow 995/tcp   # POP3S
ufw allow 4190/tcp  # Sieve

# Enable firewall
ufw enable
ufw status

Step 2: Configure DNS Records

Proper DNS is critical for email server setup. Misconfigured DNS = emails going to spam.

Required DNS Records

Add these records at your DNS provider:

# A Record - Points to your server
Type: A
Name: mail
Value: YOUR_SERVER_IP
TTL: 3600

# MX Record - Tells other servers where to send mail
Type: MX
Name: @ (or yourdomain.com)
Value: mail.yourdomain.com
Priority: 10
TTL: 3600

# SPF Record - Authorizes your server to send email
Type: TXT
Name: @
Value: v=spf1 mx a -all
TTL: 3600

# DMARC Record - Email authentication policy
Type: TXT
Name: _dmarc
Value: v=DMARC1; p=quarantine; rua=mailto:postmaster@yourdomain.com
TTL: 3600

# Autodiscover for email clients
Type: CNAME
Name: autodiscover
Value: mail.yourdomain.com
TTL: 3600

Type: CNAME
Name: autoconfig
Value: mail.yourdomain.com
TTL: 3600

Reverse DNS (PTR Record)

Configure at your hosting provider's control panel:

IP Address: YOUR_SERVER_IP
PTR Record: mail.yourdomain.com

Without correct PTR, many email servers will reject your mail.

Verify DNS Setup

# Check A record
dig A mail.yourdomain.com +short

# Check MX record
dig MX yourdomain.com +short

# Check SPF
dig TXT yourdomain.com +short

# Check PTR (reverse DNS)
dig -x YOUR_SERVER_IP +short

All should return expected values before proceeding with email server setup.

Step 3: Install Mailcow

Download Mailcow

cd /opt
git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized

Generate Configuration

./generate_config.sh

You'll be prompted for:

• Mail server hostname: mail.yourdomain.com
• Timezone: Your timezone (e.g., America/New_York)
• Branch: master (stable)

This creates mailcow.conf with your settings.

Review Configuration

nano mailcow.conf

Key settings to verify:

# Your mail domain
MAILCOW_HOSTNAME=mail.yourdomain.com

# Timezone
TZ=America/New_York

# Skip Let's Encrypt for custom certificates (usually no)
SKIP_LETS_ENCRYPT=n

# HTTP/HTTPS bind (default is fine)
HTTP_PORT=80
HTTPS_PORT=443

# API key for automation (generate random string)
API_KEY=your-random-api-key-here

# Enable webmail (SOGo)
SOGO_DISABLED=n

Start Mailcow

docker compose pull
docker compose up -d

First start takes 5-10 minutes. Monitor progress:

docker compose logs -f

Wait until you see SOGo and nginx started successfully.

Access Web Interface

Open https://mail.yourdomain.com in browser.

Default credentials:

• Username: admin
• Password: moohoo

Change the password immediately in Configuration → Admin Password.

Step 4: Configure Mailcow

Add Your Domain

1. Go to E-Mail → Configuration → Domains
2. Click "Add domain"
3. Enter: yourdomain.com
4. Set mailbox quota (default is fine)
5. Click "Add domain and restart SOGo"

Add DKIM Key

DKIM signs your emails, proving they're from your server:

1. Go to E-Mail → Configuration → Domains
2. Click the key icon next to your domain
3. Select key length: 2048 (recommended)
4. Click "Generate"
5. Copy the displayed DNS record

Add to DNS:

Type: TXT
Name: dkim._domainkey
Value: (the long string from Mailcow)
TTL: 3600

Create Mailboxes

1. Go to E-Mail → Configuration → Mailboxes
2. Click "Add mailbox"
3. Fill in:

- Username: user - Domain: yourdomain.com - Password: secure password - Quota: as needed

1. Click "Add"

Create Aliases (Optional)

1. Go to E-Mail → Configuration → Aliases
2. Add common aliases:

- postmaster@yourdomain.com → admin@yourdomain.com - abuse@yourdomain.com → admin@yourdomain.com - webmaster@yourdomain.com → admin@yourdomain.com

Step 5: Test Your Email Server Setup

Send Test Email

Using webmail (SOGo):

1. Go to https://mail.yourdomain.com/SOGo
2. Log in with your new mailbox credentials
3. Send email to external address (Gmail, etc.)
4. Send email from external address to your mailbox

Check Email Headers

When you receive the test email externally, view headers and look for:

Authentication-Results:
  spf=pass
  dkim=pass
  dmarc=pass

All three should pass for good deliverability.

Test with Mail-Tester

1. Go to https://mail-tester.com
2. Send email to the provided address
3. Check your score (aim for 9-10/10)

Common issues:

• Score below 7: Check SPF, DKIM, DMARC
• Blacklist warnings: New IPs need reputation building
• Missing PTR: Contact hosting provider

Test Email Reception

Send emails from:

• Gmail
• Outlook/Hotmail
• Yahoo

Verify they arrive without issues.

Step 6: Security Hardening

Enable Fail2ban

Mailcow includes fail2ban for brute force protection:

cd /opt/mailcow-dockerized
nano data/conf/rspamd/local.d/worker-controller.inc

Add:

count = 10;

This limits login attempts.

Configure SSL Settings

Mailcow uses Let's Encrypt by default. For custom certificates:

nano data/assets/ssl/cert.pem    # Your certificate
nano data/assets/ssl/key.pem     # Your private key

Restart after changes:

docker compose restart postfix-mailcow dovecot-mailcow nginx-mailcow

Set Up Two-Factor Authentication

1. Go to User → Two-factor authentication
2. Enable TOTP
3. Scan QR code with authenticator app
4. Verify and save backup codes

Regular Security Updates

cd /opt/mailcow-dockerized
./update.sh

# Or manually
docker compose pull
docker compose up -d

Step 7: Spam and Filtering Configuration

Rspamd Configuration

Access Rspamd UI: https://mail.yourdomain.com/rspamd

Password is in mailcow.conf (RSPAMD_PASSWORD).

Key settings:

• Actions: Configure what happens at each spam score
• Statistics: View spam filtering effectiveness
• Symbols: Fine-tune detection rules

Recommended Spam Settings

In Mailcow admin → Configuration → Rspamd:

Spam score threshold: 6
Ham score threshold: 2
Reject score: 15

Whitelist/Blacklist

E-Mail → Configuration → Filters:

• Add trusted domains to whitelist
• Block persistent spammers

Step 8: Backup Strategy

Automated Backups

Create backup script:

nano /opt/mailcow-dockerized/backup.sh

#!/bin/bash
BACKUP_DIR="/backup/mailcow"
DATE=$(date +%Y-%m-%d)

mkdir -p $BACKUP_DIR

# Stop for consistent backup
cd /opt/mailcow-dockerized
docker compose stop

# Backup data directory
tar -czf $BACKUP_DIR/mailcow-data-$DATE.tar.gz data/

# Backup volumes
docker run --rm \
  -v mailcowdockerized_vmail-vol-1:/data \
  -v $BACKUP_DIR:/backup \
  alpine tar czf /backup/vmail-$DATE.tar.gz -C /data .

# Restart services
docker compose up -d

# Keep last 7 days
find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $DATE"

Schedule daily:

chmod +x /opt/mailcow-dockerized/backup.sh
echo "0 2 * * * /opt/mailcow-dockerized/backup.sh" | crontab -

Restore Process

cd /opt/mailcow-dockerized
docker compose down

# Restore data
tar -xzf /backup/mailcow/mailcow-data-DATE.tar.gz -C ./

docker compose up -d

Step 9: Monitoring Your Email Server

Check Service Status

cd /opt/mailcow-dockerized
docker compose ps

All services should show "Up" status.

Monitor Logs

# All logs
docker compose logs -f

# Specific service
docker compose logs -f postfix-mailcow
docker compose logs -f dovecot-mailcow
docker compose logs -f rspamd-mailcow

Set Up Alerts

Create monitoring script:

nano /opt/mailcow-dockerized/health-check.sh

#!/bin/bash
SERVICES=("postfix-mailcow" "dovecot-mailcow" "nginx-mailcow" "rspamd-mailcow")

for service in "${SERVICES[@]}"; do
    STATUS=$(docker inspect --format='{{.State.Running}}' $service 2>/dev/null)
    if [ "$STATUS" != "true" ]; then
        echo "ALERT: $service is down!" | mail -s "Mailcow Alert" admin@yourdomain.com
    fi
done

# Check SMTP port
nc -z localhost 25 || echo "SMTP port 25 not responding" | mail -s "Mailcow Alert" admin@yourdomain.com

External Monitoring

Use services like:

• UptimeRobot - Free tier available
• Pingdom - More features
• Your own monitoring stack

Monitor:

• HTTPS on port 443
• SMTP on port 25
• IMAP on port 993

Step 10: Multiple Domains

Adding additional domains is easy:

Add Domain

1. E-Mail → Configuration → Domains
2. Add new domain
3. Generate DKIM key
4. Add DNS records for new domain

Required DNS for Each Domain

MX Record: mail.yourdomain.com (priority 10)
SPF: v=spf1 mx a include:mail.yourdomain.com -all
DKIM: (generated by Mailcow)
DMARC: v=DMARC1; p=quarantine; rua=mailto:postmaster@newdomain.com

Troubleshooting Email Server Setup

In my experience, these are the issues that come up most often when setting up mail servers.

Emails Going to Spam

# In email headers, verify:
SPF: pass
DKIM: pass
DMARC: pass

1. Check authentication:

1. Check blacklists: https://mxtoolbox.com/blacklists.aspx

1. Verify PTR record matches hostname

1. Send quality content - avoid spam trigger words

Cannot Receive Email

dig MX yourdomain.com

1. Check MX record:

nc -z mail.yourdomain.com 25

1. Check port 25 is open:

docker compose logs postfix-mailcow | grep "reject"

1. Check logs:

Cannot Send Email

telnet gmail-smtp-in.l.google.com 25

1. Check if port 25 is blocked by ISP:

docker compose logs postfix-mailcow | grep "status="

1. Check outbound logs:

1. Verify SPF includes your server

Slow Performance

docker stats

1. Check resources:

1. Increase RAM if needed (6GB+ recommended)

df -h

1. Check disk space:

Email Server Setup: Maintenance Checklist

Weekly

• Check service status
• Review spam filter stats
• Check disk space
• Verify backups ran

Monthly

• Apply Mailcow updates
• Review security logs
• Check blacklist status
• Test backup restoration

Quarterly

• Review user accounts
• Update passwords
• Audit email aliases
• Check SSL certificate expiry

Email Server Setup Cost Comparison

Self-hosted email server setup pays for itself quickly with multiple users.

Summary

Complete email server setup with Mailcow involves:

1. Server preparation - Clean IP, Docker, firewall
2. DNS configuration - A, MX, SPF, DKIM, DMARC, PTR
3. Mailcow installation - Docker-based deployment
4. Domain setup - Add domains and DKIM keys
5. Security hardening - Fail2ban, SSL, 2FA
6. Spam configuration - Rspamd tuning
7. Backup automation - Daily backups
8. Monitoring - Health checks and alerts

Total setup time: 2-4 hours for a production-ready email server.

The ongoing maintenance is minimal - weekly checks and monthly updates. The privacy and cost benefits make self-hosted email worthwhile for businesses and power users.

Need help with email server setup or migration from existing email providers? I can help you plan and execute a smooth transition. Let's talk about your email infrastructure needs.