We had a severe thunderstorm overnight - actually it started around 3am. The dogs would have woken me up, but there was so much thunder and the lightning strikes were so close that I was up anyway, and after I got them settled down, did what any geek would do - checked systems.
Ping tests showed all kinds of network snarls in the immediate area, and traceroutes died before they hit Cavtel's mail servers, so I figured there were probably lines down, and headed back to bed. After all, my clients call me before calling their ISP, so it was going to be a long day.
Phone started ringing right at 8 - sure enough, spikes and surges had caused some system troubles here and there, and a few folks had phone and net problems. I took my laptop with me and took notes. By 11 am connections through Cavtel's routers on the northside were ok, but there was still a lot of packet loss on the southside and I could only hit my own IP twice. Most of the trouble looked like it was sitting between 64.83.47.74 and 64.83.47.177. Since I was checking from several different locations around the city, it seemed like a pretty solid triangulation, and very consistent packet loss - above 60%, but only on the network heading towards my IP.
Given that I was still running down issues after lunchtime, and wasn't going to be home anytime soon, I took advantage of a few moments alone in a quiet lobby to start a trouble ticket with Cavtel's DSL support. Had I known how badly this would turn out, I'd have written down the guy's name.
1:30pm - Briefly - I gave him all my info to verify who I was, then told him what I'd been seeing, and asked him to start a trouble ticket. I gave him the IP addresses of the points where traceroute was failing the most often, and wrote down the ticket number - # 275681. As I was reiterating that I thought they probably had a lightning strike at or near the CO on Hull St, he suddenly blurted out "traceroute doesn't work on our servers." That should have been a major 'kloo', but I dismissed it, since I always run traceroutes on their servers when Im troubleshooting my client's net problems.
Hindsight is always so clear... From the way things went down after that, I doubt that asking for a supervisor at that moment would have changed the outcome much, but one can only hope.
Let's see, have I left anything out? Oh, yes, the nameless rep asked for a phone number so that a tech could reach me - I gave him two, and he read them back to me to be sure, said the ticket was in the system and I'd be hearing from someone as soon as they had a chance to check on the situation.