Spamhaus
Voodoo Rubber Chicken Magic
Or - Spamhaus - Part Three
Someone asked me, why don’t we just use the new reputation service in MailMarshal?
Good question.
Three reasons we decided to use Spamhaus.
1. Spamhaus is much more effective
Independent testing of live data shows Spamhaus consistently blocks about 85% of all incoming spam. Spamhaus has maintained this level of performance for a very long time. The new Marshal IP reputation service seems to block about 40%. We don’t have independent tests. What we can say for certain is Spamhaus currently blocks twice as much as the new Marshal service.
2. Spamhaus cleans your internet pipe and saves you real money
The single biggest reason to run a reputation service is to SAVE MONEY. By hanging up on the spammer when he connects to you. Average spam size in October 2009, according to our research, was 22.1 kbytes. Every single time you hang up you save 22.1 kbytes. And money. And storage. And processor. You get the idea.
Simply put Spamhaus will save you twice the money and free up twice the bandwidth and resources.
3. Spamhaus doesn’t hang up on your mother
It may sound a simple thing to run your own reputation blacklist. I suspect not. We don’t expect Marshal performance to magically shoot up to 85%. In ten years no other organisation has been able to do it. Thing is, every reputation blacklist has a problem managing false positives - that’s when it hangs up on your mother. Not good (assuming she’s not a spammer). I don’t know how Spamhaus do it, voodoo rubber chicken magic maybe, just that they do.
Spamhaus don’t have outages and don’t hang up on Mum.
4. The Marshal IP Reputation service is V1.0
Call us fickle. Call us cynical. We’re probably both, but we still like to see things prove themselves before committing.
I know I said three, but reason one could be called a summary.
In the end, the chicken won us over.
Carlton
- Rubber chickens available in toy stores everywhere.
Spamhaus arrive down under?
Part Two - Spamhaus hit town
In 2004 Spamhaus moved from a free model to a subscription model, as their chosen response to the pressures discussed in my part one post. This is explained on their web site faqs under the question Why is there a charge for this service?
We pick up the story around March 2009 when a company called MxTools started contacting a few of the largest MailMarshal customers in Australia informing them their usage of the Spamhaus services were not paid for and requesting payment or their usage would be either blocked or choked. Shortly after this time we discovered what they meant by the term ‘choke’. Without going into the technical detail, the practical results meant MailMarshal would have big delivery delays and start to queue large volumes of inbound email. In large sites this could tens of thousands of emails. These events triggered a frenzy of activity for us as we tried to get some answers from all parties as to what was going on. Aside from the queuing email, we had a bunch of wider questions, including, who is MxTools, why was this an issue now, how much will this cost and isn’t this an issue for Marshal? What we discovered was something like this.
In the years between 2004 and 2009, although Spamhaus was not free they simply did not look for licensing fees down under. Their efforts we put into the larger players in the USA and Europe. So we’d all been getting a free ride without knowing it for years. Those days were simply ending. Spamhaus themselves weren’t really setup as a commercial operations and didn’t see their core task as collecting license fees. So they had approached MxTools to act as a commercial partner so they could focus on building a robust anti spam service whilst not running out of money. Because Spamhaus only set out to cover the cost of running such a large network operation the fees were very low when compared with the costs of commercial anti spam solutions and also the data charges that were saved by using Spamhaus. The guys at MxTools assured us Spamhaus had held discussions with Marshal about entering into an OEM contract to cover all MailMarshal users. Marshal confirmed as much to us and that their position was if users wished to use that functionality they should pay any third party fees, the same as was the arrangement with other plug-ins like anti virus and anti spyware. They decided therefore not to enter any vendor contract with Spamhaus.
At this stage of the proceedings we felt very much the meat in the sandwich. Spamhaus, as represented by MxTools seemed to have a cut and dried case, our customers were using Spamhaus and it was saving them real money in direct data charges. Even if most of them were blissfully unaware as we had been. It was a feature that had been shipping as part of the ruleset of MailMarshal for years. Marshal seemed not to see eye to eye with us on this one, taking the position that if customers wanted to use third party services then they would simply pay for them as they did with their various virus scanners. Our customers wanted to know why it was their problem, and if they had paid their yearly maintenance why was anyone turning up asking for more money - to them I’m sure we smelled like a rat.
In a panic, we started to look very closely at all the DNS based services that were out there, how they worked and whether there was a alternative we could move everyone to. We looked at other providers like MAPS, SORBS and Spamcop. As we assembled the research it became apparent that all these providers were not created equal, some had block rates lower than 50% and other more like 62%, but with significant false positive rates. So they would wrongly blocks emails from legitimate sources; which we knew from years of support was a serious issue. There was also talk of what happened when these guys got attacked by spammers, which happened from time to time. This is when we discovered the depth of problems these guys had, with spammers attacking their networks on one front, and getting sued on the other.
In the end, we found Spamhaus stood out head and shoulders above them all. They had a huge robust network design, with 60 DNS servers spread over 18 countries and had never suffered an outage in 14 years. Spamhaus was famous for it’s industry low false positive rates - no one had or has a more accurate list. Their business model, although a shock when they banged on our door, looked like it was sustainable and they wouldn’t be the next DNS service failure in the world. And they weren’t owned by a large multinational, but were a true independent. In the final wash up, we wanted to be running Spamhaus with our customer base because it was the best.
So WebSecure entered into serious discussions with MxTools. We said they couldn’t just randomly start turning off our user base and causing serious email outages, the technician support implications alone were a potential nightmare. An understanding was reached that they wouldn’t contact Australian and New Zealand based customers without consulting with us and in return we would raise the issue of Spamhaus licensing as every customer’s yearly maintenance came around. Whatever the customer decided to do, either to license or not, we would communicate to MxTools and Spamhaus would turn off those customers who didn’t want the service anymore. As a result, WebSecure became a Spamhaus reseller through MxTools so we could supply the licensing in Australian dollars and control the administration.
In part three of this series I want to look at the mechanics & performance of Spamhaus and into specific setups with MailMarshal.
Where did Spamhaus come from?
Part One - Where did they come from.
The story of how we started blocking spam at the network level is, in part, a story of what happens when social conscience meets big money in the global marketplace. It starts over 13 years ago with two guys, Paul Vixie and Dave Rand, who started keeping a list of IP addresses that were sending them unsolicited, unwanted and other objectionable material in email form.
By 1996 their list had become known as the Real-time Blackhole List, or the acronym RBL, and they had founded a non for profit organization called Mail Abuse Prevention System (MAPS) and were publishing their list as a DNS based service to network managers anywhere on the internet. Using MAPS, email administrators could now use network layer DNS checks to hang up on incoming email connections from spammers, effectively blowing them off before they downloaded the email and paid for the bandwidth. Imagine the reception such a system got from the fledgling spamming community!
Rand and Vixie didn’t see how it could be wrong to publish a blacklist of known spamming IP addresses on the internet, but drew an ever stronger response as the effectiveness of the system grew. In 1998 they appointed high profile lawyer, Anne Mitchell, as Director of Legal and Public Affairs to mange the legal load they were generating. By 2000, they had had received many lawsuits threats, two of which can still be seen on the original MAPS web page entitled How to Sue MAPS
By 2001, the financial pressure of both growing a network to support the ever increasing load, and carrying the weight of legal defense changed their organization model. They began to charge subscription fees as a way to pay their bills. In 2004 the organization was sold to a commercial entity, Kelkea Inc., an anti spam vendor founded by Dave Rand. Then finally in 2005 Trend Micro Inc. acquired all the assets of Kelkea and MAPS, the pioneer of network based spam detection had ceased to exist as we knew it.
This story highlights the two biggest themes in network based spam detection.
Firstly, DNS based network blocking of spammers is an extremely effective way to get rid of a very large percentage of spam. Both stopping the spam and saving data charges at the same time. Secondly, almost all providers of these services started life as not for profit community service style organizations. Together, these two salient facts have forced all of them to change their models, either by charging for their services or entering into various commercial partnerships.
That brings us neatly to Spamhaus, which was founded in 1998 by Steve Linford using the same core technical ideas of a DNS based central lookup system for tracking spammers on the internet that Rand and Vixie had pioneered with MAPS. In my part two post I’ll explain how Spamhaus operate today and how they ended up in Australia.
