I have a Gmail address that I don’t typically publish, and a non-Google-hosted e-mail address that I do. For years I have forwarded all mail received at my public e-mail address over to my Gmail account, and for years this has worked flawlessly. That is, it worked flawlessly until sometime during the weekend of January 18-19, when I noticed that my Gmail inbox had an unusually low message count. (Fortunately, I have my server configured to retain a copy of every message it forwards on to my Gmail address, so I didn’t actually lose any mail.)

I started to run some tests. First, I tried to send myself e-mail from work, and Google bounced the message right back to me:

550-5.7.1 [xxx.xxx.xxx.xxx xx] Our system has detected that this message is
550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail,
550-5.7.1 this message has been blocked. Please visit
550-5.7.1 http://mail.google.com/support/bin/answer.py?hl=en&answer=188131 for
550 5.7.1 more information.

“That’s rarely a good sign.” Likely anyone who had tried to send me e-mail during that time period also ended up receiving a similar bounce message back from Google. In a few cases, e-mail lists that I had signed up for decided to unsubscribe me as a result.

The question, of course, was why Gmail now thought that the mail that I regularly forward to myself was spam, and why it had become a problem all of a sudden when they’ve been gleefully accepting my forwarded mail for years (3 months shy of 10 years, to be precise)?

The first thing that immediately occurred to me was that perhaps Google had blacklisted my server’s IP address for some reason. To test that theory, I tried changing the IP address to a different one temporarily (while also making sure to update DNS and reverse DNS entries), but that didn’t make a difference. As I was looking back through past e-mail headers searching for potential clues, I realized that since my server was running dual IP (v4+v6) stacks, and since Google was reachable via IPv6, that it was preferentially exchanging mail with Google by way of IPv6. Could it be that my IPv6 address was what Google had ended up blacklisting? I told Postfix not to use IPv6, tried sending myself another piece of e-mail, and voilà: it went through. I then tried giving my server a different IPv6 address (from a completely different /64 prefix, even), re-enabled IPv6 in Postfix, patched up my DNS zone files again, and sent myself another piece of e-mail. That also worked. Finally, I returned everything back to the way it was before, sent myself mail again, and bam: rejected.

This, to my mind, confirmed the theory that Google had decided to block mail specifically from my server’s IPv6 address. But for what reason? Both my v4 and v6 addresses received a clean bill of health; you’d think that if my server had been compromised and was being used as a relay by some unscrupulous individual(s) that Google would not have been the only one to notice. In any case, I reported my issue to Google since I wasn’t sure what else to do, but I wasn’t going to hold my breath.

When I casually mentioned my dilemma to a co-worker, he suggested that if I hadn’t yet published an SPF record for my domain, that I should give that a try. The interwebs backed his suggestion up: others who had encountered similar difficulty delivering to Gmail servers found that the answer lay in implementing SPF on their end.¹ Interesting, to be sure, but I had already added an SPF record to my domain’s zone some time ago. That a lack of (valid) SPF could be the problem just didn’t make sense to me either, because I wasn’t the author of the e-mails that were being bounced, I was the intended recipient. The people who had “fixed” their Gmail woes by taking the time to add SPF to their domain were having problems sending to Gmail directly, whereas I was acting as a forwarder. But I nonetheless went through the motions of verifying that my record was syntactically valid and that it was constructed correctly. And it was.

Later on, as I was once again mulling over the problem, and as my thoughts wandered toward what the exact purpose of SPF was, I had my “aha!” moment: if SPF is all about verifying that the sender is genuine, and it accomplishes this by giving the receiving MX a way to ensure that the transmitting MX is authorized to send mail on behalf of the sender, how can e-mail forwarding work within that kind of a scheme? The answer, sadly, is that it doesn’t (and it doesn’t). If my problem was at all related to SPF, it wasn’t that my SPF record was missing or invalid. The problem was that the people sending me mail were doing so from domains that had implemented SPF, and when my mail server would forward those messages to Gmail, Google would see mail that had “originated” from my server’s IP, and my server would not be a permitted sender for the return address that was listed on the envelope. So from Google’s perspective, I was falsely impersonating all of the people who were sending me mail.

As I discovered while further researching this problem, it turns out that the solution to this mess is apparently not to set up your server as a forwarder, but rather as a remailer.² That is, something which throws away the original envelope, and then generates a new one that bears your own return address instead of the original sender’s address. In such a scenario, the final recipient’s mail server would check the sender (yourself) against your own domain’s SPF record instead of the SPF record for the e-mail author’s domain, and assuming that your SPF record is constructed correctly, that check would pass. (Note that “remailing” in this way would not cause the e-mail to appear to you as if you had sent it to yourself; the value of the envelope MAIL FROM is distinct from the “From header” on the message itself, in much the same way that the return address on a physical envelope doesn’t necessarily have to match the address that the letter-writer decides to put at the top of the letter inside of that envelope.)

Remailing, though, does have some downsides. If the final mail server does need to “return to sender”, the “sender” in this case is going to be the mail server doing the remailing, and not the original sender. Which means that if the mail bounces for some other reason (mailbox is full, account has been closed or suspended, etc.), the person who originally sent the mail will not see the bounce, and so will have no idea that the intended recipient never got their message. The solution to this problem is to implement SRS on the remailing server. SRS munges the return address on the envelope so that in the event a bounce is generated upstream, the return address contains enough clues buried within it that the remailing server can use to determine who sent the original e-mail, and send the bounced message back to their address. It also supposedly does this in a secure enough manner that (in theory) the SRS munging scheme cannot be exploited by others to turn the remailing server into an open relay.

It turns out that Postfix does not natively support SRS, so one can’t just flip on a switch somewhere in the configuration. There are a handful of third-party solutions that exist for Postfix, though.³ I elected to use pfix-srsd from pfixtools. (Incidentally, these guys not only have a brief HOWTO on their site for installing pfix-srsd, but they also wrote an excellent description — complete with illustrations — of this very problem.)

Adding pfix-srsd to my Postfix server cured my forwarding-to-Gmail problem instantly. Immediately after setting it up, I sent myself more test messages, and they all came through. I temporarily reverted the changes and tried to send myself more test messages, and those were bounced by Google in the same manner as before.

Mail that I am forwarding to myself now contains stuff like this in the header, which gives me warm fuzzies:

Received-SPF: pass (google.com: domain of SRS0=XxXx=XX=sender.domain.com=sender-username@remailier.domain.com designates 2001:dead:beef::face as permitted sender) client-ip=2001:dead:beef::face;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of SRS0=XxXx=XX=sender.domain.com=sender-username@remailier.domain.com designates 2001:dead:beef::face as permitted sender) smtp.mail=SRS0=XxXx=XX=sender.domain.com=sender-username@remailier.domain.com

The bit that still puzzles me about this whole thing, though, is why my method of doing e-mail forwarding to Gmail worked just fine for forever and then seemingly — and without warning — became an issue almost overnight. I decided to go back and look again at the headers of e-mail messages that I had successfully forwarded to myself prior to all of this, and here is an example of what I saw:

Received-SPF: fail (google.com: domain of sender-username@sender.domain.com does not designate 2001:dead:beef::face as permitted sender) client-ip=2001:dead:beef::face;
Authentication-Results: mx.google.com;
       spf=hardfail (google.com: domain of sender-username@sender.domain.com does not designate 2001:dead:beef::face as permitted sender) smtp.mail=sender-username@sender.domain.com;

…and yet despite this, Google dutifully delivered these messages to me, didn’t bounce them back, and didn’t apply the Spam label to them. We also know that it can’t be the case that Google decided only within the past couple of weeks to start rejecting mail globally if it failed SPF. How do we know this? Because…

1. Others who run MXes have had something similar happen to them, and just as suddenly, but they experienced this issue months — sometimes years — ago, well before my IP address began to face similar discrimination.
2. I proved to my own satisfaction earlier that just by changing the IP address my server uses to speak to Google, Google would resume delivering mail that I forwarded to it even when I didn’t have SRS remailing enabled on my server. So the source IP address is definitely a variable.

This means that Google can’t possibly be applying the same SPF policy to all sources of mail across-the-board. When receiving e-mail from some IPs, Gmail does not necessarily take an SPF fail into account. (Perhaps this is the default policy.) But for other IPs, Google will opt to block any mail that it receives unless SPF explicitly passes (or, presumably, the DKIM signature checks out).

Why does Google treat mail from some IPs one way and mail from other IPs a different way? I haven’t been able to find a single document anywhere in which Google lays out what their exact policies are in a comprehensive manner. If I had found such a document, I could have been spared a lot of grief. It would also be nice to know what specific event caused Google to change how they treat my mail server’s IP specifically, so that if my server has been compromised, or if there is something else that I’ve done wrong that I am overlooking, I can work to correct it. The closest thing I’ve managed to find from the horse’s mouth is this. Perhaps they feel like they need to hold their cards close to their chest so that they don’t accidentally reveal to spammers any potential holes in their policies that might be possible to exploit or work around.

Ultimately, the lessons I took away from my own experience are these:

1. If you haven’t already created an SPF record for your domain, do so.
2. If your mail server allows its users to forward mail off-site, then implement SRS remailing on it ASAP.

As a mail server admin, you need to do these things not merely to ensure that your legitimate mail is always being delivered to Google successfully, but because Gmail has demonstrated that other mail server admins can and do enforce SPF policy selectively, and you have no idea how other mail servers might be configured to respond to SPF policy failures.

For many years, this cartoon epitomized my view of most blogs. My goal is to ensure that this one avoids the fate of becoming a caricature of such sites.

About

Hello there. Name’s Nathan. Nice to meet you. I’m a lifetime tech enthusiast and wannabe/amateur engineer. “Wannabe” because even though I’m employed within the computing industry, I have received no formal training, and the more times I’ve been around the block, the more I’ve come to understand just how pitifully little I actually know. This can be discouraging, but mostly it just makes me thirsty for more opportunities to acquire experience and to further my understanding and my craft.

Whenever my knowledge has come up short and been found wanting, I’ve relied heavily on the broader tech community on the Internet to bail me out. And I feel that it’s high-time that I started giving back. This site is my way of saying “thank you” to anyone out there who has taken the time to document the solutions to the technical problems they have run up against. I have directly benefited from the efforts of countless such people. I aim to publish my own discoveries and solutions here to some of my own technical puzzles, and my hope is that other people who have encountered some of the same problems that I have will find at least part of the answer here.

Comments

As you may (or, more likely, may not) have noticed, this site lacks a commenting feature. That’s not because I am actively trying to discourage dialogue and feedback from others about what I write. Far from it. If you have some thoughts about something I’ve posted here, please post it somewhere and contact me so that I can link to it!

Credits

Speaking of benefiting from the work of others, credit where it’s due: this site is powered by Second Crack (with some select patches from Continental folded in), and the theme was modified from and heavily influenced by Marco’s. Graphics were made in Microsoft Image Composer, for which I make no apologies.