Client Gets Blacklisted By Their Customer's System, After Customer's System Sends Bad Emails!

I have a client whose customer's Websense anti-spam server was blacklisting them. No amount of convincing would get the techs at my client's customer to whitelist my client.

That meant we had to figure out, with limited information, why we were getting blacklisted. The techs weren't very helpful. They just kept saying, our server is seeing your backscatter and you won't drop off until 24 hours after you stop spamming us...

To help make the following explanation a little clearer, when I say CLIENT, I mean my client. When I say CUSTOMER, I mean my CLIENTs customer.

The problem turned out to be multi-leveled. For one thing, a rogue computer had been using the CLIENT's network and had been infected. Because of open policies on the outgoing firewall (we had to allow port 25 for some very important customers that would work onsite on the CLIENT's network), this infected computer ended up putting the CLIENT on a number of blacklists.

This was fairly easy to resolve as we implemented a rule on the Sonicwall firewall to block outgoing port 25 traffic, except for the Zimbra and Barracuda servers. We also tracked down and fixed the offending non-CLIENT laptop. As an added measure, the wireless network used by the uncontrolled non-CLIENT laptops was separated out and put on a different network and IP address so that anything that happened there was not attributed to our domain/IP address and, thus, we wouldn't be blacklisted. (And thus we were able to successfully get the barn door closed after the horse was out!)

A little more work, testing, and follow-up and we were off the blacklists.

At about the same time, an unrelated CLIENT blackberry user was added to the group list on the Zimbra server that a certain CUSTOMER would automatically email to, using their SAP system and sending from a user id of sapemail@customerdomain.here

For some reason, the CUSTOMER SAP system automatically emailed my CLIENT a PDF file and marked the email "Request a delivery receipt for this message". That might make sense in and of itself, but the email address in the "Return-Receipt-To:" and/or "Disposition-Notification-To:" header fields was always an invalid address! We couldn't get the CUSTOMER to fix this.

Here's where it gets really weird...

When the CLIENT email system would reply with a delivery notice, there wasn't a problem. The CUSTOMER system seemed to expect this.

However, since this user was also a blackberry user, when the blackberry network picked up the message and forwarded it to the phone, it would respond with a delivery disposition message from the blackberry network. There is no way for us to configure this AT&T blackberry phone to stop doing this. So, it seems that since the delivery disposition was returned to the originating system with an invalid email address (sapemail@) that it provided and since the FROM: address was from the CLIENT domain, but the SMTP server itself was from the blackberry network, the CUSTOMER's Websense scanner was considering the message as SPAM (backscatter) and the CLIENT domain was blacklisted for 24 hours.

Sheesh.

Here's a step-by-step of the sequence to help clarify the process:

1. Auto email sent from CUSTOMER to CLIENT with bad REPLY TO email address sapemail@ and DELIVERY NOTIFICATION set on
2. CLIENT sends DELIVERY NOTIFICATION to invalid email account sapemail@
3. CLIENT's blackberry account picks up email
4. blackberry server sends DELIVERY NOTIFICATION to invalid email account sapemail@
5. CUSTOMER's email server receives DELIVERY NOTIFICATION from CLIENT, no big deal
6. CUSTOMER's email server receives DELIVERY NOTIFICATION from blackberry, tags CLIENT domain as spammer and blacklists them. May or may not tag blackberry network as spammer.
7. CUSTOMER's email server refuses all email from CLIENT.
8. Repeat at least 10 times a day...

So how did we fix it? We think the CUSTOMER whitelisted the CLIENT because we are no longer blacklisted, but we haven't stopped the rest from happening. If it happens again, we'll either put the blackberry user on a different email address for the sapemail@ emails or we'll have to try negotiating with the CUSTOMER again. I think the CUSTOMER may have found out that other companies besides my CLIENT had this issue and either reconfigured the Websense box or just whitelisted us. Or maybe the blackberry SMTP server was whitelisted because others were getting blocked there... Just not sure right now. I'll update this later, if we find out more.

Update 12/18/2008:
I found some options on the email account setup on the blackberry that might have helped. On the blackberry Curve, I did this:
* go into email program
* press menu button
* select Options
* select Email Settings
* change "Confirm Delivery:" to NO