Tracking down an accidental Gmail DoS
Sat 30th Nov 13
If anyone was watching my Twitter feed over the last few days you'll have seen me complaining about my Gmail account being down. It wasn't down completely, I could still access the web interface and read all old mails but hadn't had any new emails in since 4AM on Thursday. I have various other mail accounts, some Gmail, some not, so I tried sending myself mails from those account to see if things were broken or whether I had just become very unpopular. None of the mails got through. I also tested sending emails out and none of those worked either so there was definitely a problem. By Friday lunchtime I'd had a couple of mails but nothing much so I figured I'd better do some digging and get it fixed.
My Gmail account is actually a Google Apps account, one of the free variety that I've had for years. Google are offering a free 30 day trial of their paid for service which includes 24/7 tech support so I decided now was the time to sign up for it. So I signed up, gave it five minutes for the account to become active then called them up. There was no comment that I'd just signed up and the tech guy was happy to help.
The guy told me there were no current issues with Gmail so it must be something specific to me and my account. He had me log into the admin console and look at the email report log. I'd not seen this before and to get to it you need to select "More controls" at the bottom of the console then "Reports". In there select "Email Log" from the left hand menu.
There are various filtering options available so I looked at the last couple of days and instantly saw the problem. The report showed a page with about 100 emails, all of which were marked as status "In progress". It also said that that was page one of about 10,000. All the emails were from the same address, one that I set up to send me reports of bugs on a site I used to manage.
The mails were being sent at the rate of between one and five per second, from 4AM Thursday to 4PM Friday that is a lot of emails.
At the same time this was happening a friend got a bounce from a mail he had tried to send me and passed it on to me, the reason given was:
1.1.1.1 does not like recipient.
Remote host said:
550-5.2.1 The user you are trying to contact is receiving mail at a rate that
550-5.2.1 prevents additional messages from being delivered. For more
550-5.2.1 information, please visit
550 5.2.1 http://support.google.com/mail/bin/answer.py?answer=6592
Things now made sense, the old site, for some reason, was flooding me with emails at such a rate that Gmail was refusing to accept any more mails.
As I no longer maintain the site that was sending the mails I couldn't just shut it down. I talked this through with the Google tech who gave me two ways to block the mails. The first was to block the sender completely.
To do this, go to the Google admin console and select "Google Apps" then "Gmail" then "Advanced settings" at the bottom. In this page, look for "Blocked senders". In here I entered the email address of the sender but when I tried to save it I got errors. It said I'd not filled something in correctly even though I thought I had. The tech told me that this is a known issue, refresh the page, try again and it will work. It did.
The second option, just to make double sure, was change the routing for mails from that address. This is done from the "Default routing" option on the "Gmail" page. Here I entered the sender's address again and told it to reject the mails.
It has taken a few hours but mails are now slowly coming in, I'm sure I'll have missed some, hopefully I'll get most back but I know people have received bounces. I've contacted the hosting company of the site that is hammering me and they have removed email privileges from the affected site so hopefully I'll get no more mails. I've no idea why the site is broken and will probably never find out.
So, why this blog post? This was an accidental DoS on my mail and one that took a bit of effort to track down. I've never heard of this happening before so thought I'd write it up so that if it happens to anyone else it won't take them as long to fix it. Also, as the mails were sent from a quite basic hosting account it shows that it doesn't take much to take out a Gmail account. This could easily be used by the bad guys as a deliberate DoS. With the reliance on email, if your main account was disabled for a day or two how would you cope? Do you have a plan in place? I didn't but I am going to think about setting one up now.
Finally, the £33 per year the Google subscription is going to cost me is definitely worth it, I figure I got my moneys worth out of it with this one call. The steps above actually took about two hours and rather than stay on the call through it all Bryan, the tech, called me back each time he needed more information or had a suggestion. This is about the best support I've had in a long time, certainly much better than I'm getting from the solicitor sorting out my house move who is charging a lot more than £33.
Update: I got a mail from Bryan on Sunday saying he wasn't working again till Monday afternoon but wanted to check that things were getting better and to tell me he would be back on the case when he got in. That is good customer service!
