Offering technical support frequently feels like being in a detective story. The phone rings and you know someone is in trouble, and that they need your help to figure something out.
Our story begins under just these circumstances.
Tim O’Shea, our resident Systems Samurai, made a number of improvements to a business client’s setup, replacing the router and setting up a NAS device for file sharing.
Soon thereafter, the phone rang.
“Their ISP sent them graphs,” says O’Shea. “They were maxing their bandwidth.”
In simple terms: they were using up all of their Internets, quickly, and didn’t know why.
I gets weirder. Almost all of this new traffic was outgoing, which is odd because most businesses download way more than they send out. In this case 80 percent of all traffic was outgoing, meaning a tremendous amount of information was flowing from their office to somewhere else.
“Long and short: you should see more down than you see up,” says O’Shea.
Something very weird was happening.
O’Shea knew none of the changes he recently made caused this; the router and the NAS device both function inside the network, meaning neither should be accessing the Internet.
This meant the problem was almost certainly happening at one of their computers.
“This is an office of 20 desktops,” says O’Shea, “Most of them Macs.”
So it probably wasn’t malware.
Not sure what the problem could be? Neither was O’Shea. So like any good detective he searched for more clues.
Routers track traffic, leaving behind a trail for O’Shea to follow. But this led to even more confusion.
“The breakdown showed it wasn’t SMTP or HTTP traffic; it was ‘Other,’” said O’Shea.
Confounding.
Further research found the outgoing data was going to an email server: Google’s, to be precise. This meant someone was emailing out gigabytes of data every day, or something was going wrong with someone’s email set up.
Through further examination of the logs, an a bit of trial and error, the culprit was eventually found.
“It ended up being the front desk,” says O’Shea. “The receptionist had three emails in her Outlook outbox with 36 MB attachments trying to be sent via Google, which has a 20 MB limit. It kept trying to send over and over and over again.”
Sound like a simple fix? It is once you know what the problem is. Figuring this out takes time, though.
“One desktop was saturating their bandwidth,” said O’Shea. “Figuring this out took the better part of two days.”
What’s the lesson? Well, first of all, information is good. O’Shea solved the mystery because he had access to clues.
“The information allowed me to figure this out in a day and a half, versus a week without it” said O’Shea.
The other lesson? Problems aren’t always obvious.
“In many environments it’s easy for one person to, purposefully or not, have a significant impact on resources in IT,” said O’Shea.
Unravelling these problems isn’t typically easy. That’s why we do what we do.
Do you have a technology mystery that needs solving? Contact us. Chances are we’ve run into something similar before.
