I woke up today and went to check my email as I normally do, only to find my mail server completely unresponsive. It's a VM running under Proxmox, so I logged into the admin interface and found all the CPUs pegged at 100%. Still, I could log in and quickly spotted what you never want to see: a bunch of unknown processes taking up all the CPU cycles. A quick search revealed the worst—the process sft was some crypto-miner; my email server had been compromised.

I've always said that more people should be like Hillary and run their own email servers out of their basements. But, naturally, with that comes the responsibility of keeping things updated and secure, and on those fronts, I admit I could have done better.

History

I've been running my own email server since 2000. Back then, you typically cobbled together the pieces you wanted: an SMTP server, POP (as was common), and IMAP (which was new and fancy but more resource-intensive). Webmail clients would come later; back then, Pine was all the rage. The internet probably would be a better place if it stayed that way.

But over the years, I eventually stumbled upon the Zimbra Collaboration Suite, which offered both an open-source "free" edition and a paid Enterprise suite. It was more than enough for my personal needs: web-based administration and email client, with all the services you would expect—and more tightly integrated, including contacts and calendaring. While overkill for my personal domains and dozen or so email addresses, it provided good experience, especially when I later got a job at a company running ZCS internally for hundreds of employees. Being familiar with ZCS certainly helped.

Of course, nothing lasts forever. Zimbra EOL'd the open-source version 8.8.15 toward the end of 2023. I knew I would need to move because, while I loved using Zimbra, the Enterprise version was far too expensive for personal use. I looked over my options at the time and tested a few; nothing really stood out immediately. I figured I didn't have to rush.

Overall, it wasn't terrible that I kept running it for so long. I had several things going for me that kept things pretty secure—again, this is running in my home basement.

My setup:

• Zimbra ran internally under Proxmox, with replication and regular backups

• I use VLANs pretty extensivly in my home network and have a dedicated DMZ network where Zimbra lived in isolation

• I use MX Guarddog to block spam and other malware—I can’t recommend it highly enough. As a bonus, this means SMTP is the only external port I need to forward to the Zimbra server, and I could further restrict that flow only to MX Guarddog's known IP addresses.

• Only my internal trusted VLAN could access IMAP, webmail, SSH, or anything else. I can VPN to my home network if I needed to get to mail remotely.

Sadly, I have to deal with Verzion, which blocks outbound SMTP traffic. This meant I had set up Zimbra to use SMTP2GO for all outbound delivery. And that system had worked pretty reliably for many years.

The Incident

Two weeks ago, MX Guarddog suffered a total outage when their data center's cooling systems failed and they were forced to shut everything down. While I was surprised they were single-homed like that—and that a data center failure could cause such an extended outage—I had been so happy with their spam blocking ability, and given this is the first major outage from them I remember noticing (I've been using them since Google killed off Postini back in 2012).

Since it wasn't clear how long this outage would last, and because I needed to receive Multi-Factor Authentication (MFA) logins, I figured I would be okay pointing my MX records at myself and temporarily opening Zimbra's SMTP port to the wider world. (Foreshadowing: It did not turn out okay.)

So, I made the DNS changes; email started flowing again, and things were good. By the end of the day, MX Guarddog had their data center back online, and I switched my MX records to point back to them. I thought I had put all the firewall rules back in place, but apparently, I missed one crucial rule...

Getting Hacked

That's the thing with security: Running old or unsupported software isn't a problem if it's isolated and firewalled from bad actors. But once you open that door even a crack, you better assume not just that bad actors might get in, but that they will. I was impressed it took as long as two weeks for the breach to occur, but leaving the door open a crack is all it takes.

I do want to be clear: this setup was for my personal home lab where the stakes are low. In a professional or corporate environment, you always have the money and resources to keep software patched and up-to-date so that kind of stuff never happens. (Okay, that was a joke, but in that environment, far more thought and planning goes into making changes like firewall rules than applies here.) I’m writing this up to document my learning experience, hoping others can do as I say, not what I do.

Two weeks later, no email access. Logging into the server immediately flagged something strange. An unfamiliar process was consuming all the CPU. A quick query of Claude told me immediately that it was a cryptomining payload, likely resulting from a Zimbra exploit.

zimbra     19058  564 19.7 3043152 2409008 ?     Sl   11:00   3:11 ./sft -c /dev/null -o 167.172.176.XXX:2222 -u 83sgNtC4Fxgf5SHrFKcgaabku5wH8jruJ7Zkp5mVcSR4Np24RSRi7s1Q5j7aSk8eTjdFrpx57D3KuNdbFSQ24xZaRTaVtpp -p zimbra -t 6

Claude was even able to point me to CVE-2026-73570, an actively-exploited Zimbra flaw involving the zimbra-snmp package. It looked exactly like what I was dealing with.

At that point, it was clear: my years of living fast and loose with this ancient version of Zimbra were over; I had to move platforms. First, though, I needed to stop the current bleeding and assess the full extent of the damage.

Since Zimbra was running on Proxmox, my first action was to disconnect the network adapter completely to fully isolate the machine. On my router, I found that SMTP was still open to the world instead of being limited just to the MX Guarddog IPs—that was how they got in.

I also identified roughly when the attack started (or at least when the mining began). If I could patch the hole, I might have had the option of rolling back the server a day or two (again, personal email server; losing a day or two of history isn't the end of the world). But I couldn't be 100% sure they hadn't gotten in sooner and poked around before launching the obvious payload. In these cases, you have to assume everything on that machine is compromised: data has been stolen, every password needs changing, keys must be rotated, etc. The only safe course of action is a full machine rebuild and migrating the data off as safely as possible.

Analyzing the Options

I searched for any patch for my specific version of Zimbra or in the community. I did find Maldua's Zimbra FOSS, which looked promising and actively maintained—it would keep me on familiar ground. But honestly, I was already two major revisions behind (and their "How to upgrade" docs were still listed as "TODO") while I had years of platform experience ready for something simpler and, dare I say, more "modern."

I went back to Claude, told it about my needs, history, and what I needed in a replacement. Key factors: easy to use/maintain. It gave me three options: Mailcow (dockerized), Carbonio CE, and Maldua Zimbra FOSS.

Carbonio CE was interesting since I was familiar with the Zextras team, and it’s also a fork of Zimbra, but ultimately, it was still basically the same.

Mailcow looked to be easy to maintain yet still feature-rich: webmail, spam filtering, anti-virus, Dovecot. While I shouldn't need things like Netfilter or Unbound, it was nice to see them included. Having it run entirely in Docker initially gave me pause—I know all the kids love Docker, but I’m still old school and spinning up a dedicated VM or LXC isn't a big deal. However, since I already run a self-hosted Kubernetes cluster in my home lab, I'm familiar with running containers; I just don't often do full Docker deployments. On quick skimming of the documentation though, it looked like the installation would be simple (and more critically, easy to maintain). Furthermore, all important data volumes were straightforward to access. Best of all, Mailcow uses Maildir to store its mail, which is a huge plus over Zimbra (and I probably even miss using Pine!).

mailcow install

Since I already had an Ubuntu 24.04 template in Proxmox, spinning up a new VM was seconds fast. All I had to do was set the VLAN to my DMZ network and give it a static IP address (keeping the DMZ isolated means not exposing the DHCP server).

The Mailcow documentation was fantastic; everything was provided on one page. The basic steps are:

Install dependant system packages.

Install Docker/Docker Compose.

Clone the mailcow repo.

Generate a config file.

Start mailcow (via docker compose).

Easy peasy lemon squeezy. It was actually really easy. My only complication was that I use DNS RFC2136 with Let's Encrypt for all my domain verification (and you should too! I’ve been meaning to write a blog post about how great this feature is), but it looked like Mailcow couldn't handle that automatically out of the box. Luckily, the documentation had an Advanced SSL page detailing how to use your own generated certificates.

I set SKIP_LETS_ENCRYPT=y in the mailcow.conf file and manually configured Certbot to interact with my RFC2136 server and credentials.

Since my TSIG key was on the Zimbra box too, I had to assume it was compromised. Generating new keys was a necessary and timely measure.

I created a renewal hook that would automatically copy the certs and restart the Mailcow services:

/etc/letsencrypt/renewal-hooks/deploy/mailcow-deploy

#!/bin/env bash

cp /etc/letsencrypt/live/mail.bwi.axcella.net/fullchain.pem /opt/mailcow-dockerized/data/assets/ssl/cert.pem
cp /etc/letsencrypt/live/mail.bwi.axcella.net/privkey.pem /opt/mailcow-dockerized/data/assets/ssl/key.pem

Restart services

docker restart $(docker ps -qaf name=postfix-mailcow)
docker restart $(docker ps -qaf name=nginx-mailcow)
docker restart $(docker ps -qaf name=dovecot-mailcow)

With the certs in place, started things up:

docker compose pull
docker compose up -d

and logged into the admin UI.

The dashboard showed all containers were running. The first thing was setting a new admin password; this was initially confusing because I expected clicking the admin username in the top right to lead to user settings, but it just logs you out of the admin interface. Turns out, you have to go to System -> Configuration -> Access and edit the user from there to set the password. I really liked that they included a button to check haveibeenpwned.com to see if your password was known to have been exploited—this is going to be my kind of software! I went through all the various configuration options available. Things are divided between "System" for system configurations and "Email," where all the domains, accounts, and management happens.

I created a test domain and my first account (they call it a Mailbox). Simple and straightforward—no surprises. It looks like I could integrate with my LDAP/Active Directory server for user accounts, but that would mean exposing my domain controller to the DMZ, which is not worth it for less than a dozen accounts. Furthermore, I had to assume all account passwords on the Zimbra server were compromised, so new passwords for every account were required.

I had originally assumed I would need to use a tool like Imapsync to migrate emails over to Mailcow, so I was very surprised to see the "Sync Jobs" menu that lets you sync from another server built right into the admin UI. After removing all forwarding rules to the Zimbra server and clearing out the cryptominer, I reattached the network interface to the VM and brought the Zimbra services back up to run the sync.

It took a few minutes to start, but once it was going, it seemed to proceed pretty fast. My 26k messages of email history, stretching back to 2000 (about 2GB), was viewable in the Mailcow web client. Folders and attachments were all in place.

Final Setup

So far, everything looked great, so I decided to go all the way and have this thing start accepting and sending emails.

I recreated the firewall rules to forward SMTP traffic from MX Guarddog to the Mailcow VM, making sure to only forward traffic from MX Guarddog's known IP list. A quick click of the test button in MX Guarddog verified that it was working, and I received the test email in my Mailcow account.

The last step was configuring SMTP2Go for outbound mail delivery. This was done under System -> Routing where they call it a "Sender-dependent transport." I could either set this globally or by domain. I went with by domain, as I figured I might want to use a different service for one particular domain in the future. I created a new username/password in SMTP2Go (and deleted the old one Zimbra was using), connected the SDT to the specific domain record, and confirmed everything worked with a test email.

The desktop and mobile email clients were then re-pointed to the new IMAP server without difficulty.

In Conclusion

I'm pretty surprised at how quickly I was able to migrate my old email infrastructure over to Mailcow. The whole process—from initial install to mailboxes sync'd and email flowing in both directions—took only 1–2 hours, and most of that time was spent poking around or reading documentation. For some reason, I had built it up in my mind as going to be a long, arduous project, but it was anything but. Of course, having all the necessary infrastructure in place certainly helped! If you've been stalling on migrating off an unsupported legacy platform, don't wait any longer!