People have died for less.
Thursday I arrived at work around 7:45 AM, the usual time. I swap the backup tapes, read a few server logs, and check the voice mail. Around 9:30 AM roommate calls. Her car has overheated on the way to work and she needs me to come pick her up from the mechanic….and thus began possibly the worst 36 hours of my professional life.
I pick up roommate at the Arco station, drop her off at home and begin making my way thru late morning Los Angeles traffic. Now I remember why I start work at 8 AM. It allows me to miss all this fun. I’m about 15 minutes from the office and Boss calls. She can’t get on the Internet This is a frequent occurrence as our Netscreen firewall has a 10-user license but we have 15 computers. However, we only have 9 employees so it’s usually easy to find unused computers to shutdown. Actually I purchased an unlimited user license from CDW earlier this week; it just hasn’t arrived yet.
I tell the boss to shutdown one or both of my computers and I”ll settle it once I get back. Five minutes later she calls back to say not only did it not work but now she can’t get email either, nor can several other people. This is unusual but I’m almost there so help is on the way.
I get back to the office and check each computer one by one. Four computers have access to the internal network but not the WAN. This definitely a new. According to the logs the firewall seems to think people are getting out, but they aren’t. There are no license limit violations being logged. For seven computers everything is fine; for the rest all is not well and unfortunately one of those is the mail server. That means NOONE is getting mail in or out. It also means our self-hosted web server is down. I reboot the firewall, DSL modem (don’t start on this) and still no go.
I’m now four hours into the outage and I’m at my wits end. I call my ace-in-the-hole consultant OC-3 (who got me this job in the first place) and ask him to come take a look. Wouldn’t you know it? His wife just delivered her second child today so the soonest he can get here is Monday. An hour later he calls back and offers some time troubleshooting over the phone while he’s waiting. We dissect things for a couple of hours and get nowhere fast. Finally he suggests getting a Linksys router just to see if it will work; that way we will know if it’s the firewall, the modem or the line itself.
At 4:30 boss gives me the okay to run to Compusa. I should be back in no time flat……WRONG SUCKA! Nothing in L.A. involving public roads happens fast. As it turns out today of all days Century Blvd , which we are on, is shut down 3-7pm for protests/marches by hotel workers. I discover and alternate route and manage to sneak out the back way. On the way back I discover nearly EVERY road within a half mile radius is closed. I drive around in circles until I find a way back to the building. It took me 90 minutes to go five miles and back.
It’s now 6:30 PM and I’m starving since I worked thru lunch. I grab my food from the fridge and wolf it down. I decide to make a run to the restroom before I settle in to work. The restrooms on the 6th floor are out of service due to renovation on the floor so I have to upstairs to handle my bidness. I get on the elevator, push 8 and immediately zone out. Next thing I know the door opens on floor #5. WTF…the only floor I can get to is the lobby! It dawns on me that it’s now after 7pm. That means the elevators are now secured. That means I have to swipe my parking garage card in the elevator before I can get to any other floor. Oh well fuck me! I can’t get my parking card because my keys are upstairs and I can’t get upstairs without my parking card. Hmmmm. That isn’t a mouse in my pocket and there’s nobody I’m happy to see so what is this bulge in my pants. Suh-weeeeeeeet! Car keys! Luckily because I’ve just driven somewhere I have my car keys in my pocket. In normal instances they’d be in my desk drawer and I would’ve been SOL.
I get to work configuring the Linksys and still no go. I shutdown the server, remove the firewall, configure my workstation to use one of our unused public IP addresses…still no dice. I fiddle around with things until 10:30 PM or so and surrender for the night. I’ll get in an hour early tomorrow and surely I’ll have something working by the time people stroll in at 8 AM.
I make the hour long drive home ready to collapse onto my bare mattress. I had removed all the sheets because the dog spit up on my bed the day before and I hadn’t put clean sheets on. I arrive home to find roommate has put clean sheets on for me…an apparent attempt to butter me up for a ride to get her car in the morning. Shit shit shit! So much for going in early.
The next morning we leave at 8AM to drop her off at the mechanic. On the way boss calls to say he’s arranged for a friend/computer consultant to call in at 8:30 and asks if I’ll be there by then. Sadly I won’t so the opportunity is lost. Fuckeroni!
I get to the office at 9 MA and OC-3 calls to help me some more. While we’re on the phone Mike (the guy who had my job before me) calls to ask something unrelated. He asks how I’m doing…big mistake…I unload on him. As luck would have it he has a client that had an issue very similar to this two weeks ago. It seems the ISP made a change to the network that the Netscreen didn’t like. They ended up buying a new Netscreen then upgrading the firmware to an even newer release.
After conversing with the boss we agree this sounds like our ticket to the promised land so we arrange to buy a new Netscreen and have it sent over by courier by closing time. With any luck I can have it working without having to stay more than a couple hours over. To make the deal even sweeter Netscreen has changed the pricing on new models. The new one we just ordered already comes with an unlimited user license AND it’s cheaper than the $700 CDW.com was charging us just for the license upgrade on our existing Netscreen.
At 3pm the vendor calls to tell us the courier is stuck in traffic and he hasn’t even picked up the hardware yet. He’ll never make it today. Boss arranges to have it delivered to his house instead so I can come in tomorrow and get it running. Time out, biatch! It’s Fall. That means college football. Oh hay-ell no! If you think I’m spending Saturday working when my Georgia Dogs are on national television you have lost yo mind. I can’t let email and web presence stay down til Monday so I resign myself to coming in.
Sitting at my desk sulking I decide to call SBC, our ISP. I hate doing this because I never know which number to call. We have had our account a long time and they have merged, split and changed names more times than I can count. Departments close, numbers change, etc, etc.
I call the first number I have, wade thru the menus and give up and I get asked about Yahoo ID’s. This doesn’t sound like the right area. I call the second number, wade thru more menus whereupon the woman asks me about Covad account numbers. I call the third number, wade thru more menus and finally get a human. Of course he’s in India. I immediately experience Vietnam flashbacks of past sales calls from Dell Sales people from India. We go thru some gyrations and he passes me on to his upper level support.
Ooh, she sounds nice….translation = English….yes, minorities can be bigots too. Sue me. A few minutes later I realize she’s also Indian after this exchange:
Her: I want you to allow remote connections to your modem temporarily.
Me: (ugh)….done
Her: Can you give me the admin password?
Me: (double ugh) R-E-D…
Her: our ID?
Me: huh?
Her: our ID?
Me: what ID? you asked me for the password
Her: yes, what is the password?
Me: ooh (she was repeating r-e-d to me) r-e-d blah blah
* obviously I changed this and disabled remote admin before posting this blog, knuckleheads.
a minute or so of silence whiles she talks under her breath
Her: it should be fixed now
I test it…hmm seems to be working. I hear someone down the hall cry out “I got an email!”. Another cry is followed by others still. Remember, email has been down for 29 hours at this point. Email is flowing and the website is up! My job is saved and college football is on! 15 minutes before closing time too…..sweet! Later I hit Outback Steakhouse for a couple of 22oz Fosters and some buffalo wings.
P.S. India r0cks!