3800 THIS IS MADNESS!!!!!

Fish

From the quiet street
TCG Premium
Aug 3, 2007
40,517
7,871
Hanover Park
Real Name
Fish
So Im doing my usual 12 hour overnight shift. Well I also only got a 4 hour nap this afternoon. :lolsign: Ohh well. Sure to be a slow night. HAHAHAHAHA ROFLMAOWTFBBQOBGYNDBZ!!!!!

So at around midnight I lose everything. My screens go dead, lights go off, emergency lights go on and the fire panel starts beeping. Probably cause we lost power. :blink: Well, 10 seconds later our huge fucking generator that sits outside and does nothing for 99% of the time its there kicks on and boom. We have power back to our building. Awsome, it did its job. My screens come back on and all 21 PCs in this room are still active and online. Great, the $9000 in UPSs and batteries we spend did their job. Not enough room to power all the monitors but in theory as long as the backup generator kicks on that should only affect me for 15 seconds. Well, thats what happened. Great.

Well I get the ball rolling and let everyone know what happens. I get called by IT asking to check both computer rooms to make sure they are online and cooling systems still activated. They are. Great. We lost no important info and we shouldnt have any hicups come monday morning due to data loss. Com Ed fixes the situation and boom. The generator goes off. All is well. BUAHAHAHAHAHAHAHA!!!! Only the start of my night. And by the time this is all done its 0130.

Then I get a phone call at 0350 stating that one of our computer rooms is having a high temp alarm. That cant be. I checked them when the power went out and they both were fine. I go back to the data centers and one of them is cool and normal. I goto the next one and the lights are out, equipment still on and our cooling units are down. Its 94*!!!!!!!! Definatly not good. I get a maintenance manager on the phone and he walks me though resetting some sort of circuit unit that controls power that comes from a second huge generator that powers our # 1 and 2 warehouse. Thats right. We have roughly a 50000 SQ foot office and half of one warehouse run by 1 generator. The other half and a complete other warehouse run by another generator and a third generator to power our third warehouse incase of a power outage. Our third warehouse never went down since it is on a totally different grid. At least theres some good news there. So I reset the circuit that controls the power from the second generator and wala. We have power going back to the cooling system and lights in our secondary data center. Temp starts dropping before I even walk out of the room so thats a great sign. Another situation evaded. Thats 2 in one night. Normally we each get one about once a month. I just delt with 2 in one night. All is well and I can start to relax again. HAHAHAHAHAHAHAHAHAHAHAHAHAHA!!!!!!!

I get a call from someone else in IT asking the status since no one let him know that I had just taken care of situation #2. He tells me that he needs me to check some networking closets and a UPS room since they all have their own small cooling systems and doesnt want anything else to get damaged or run the risk. Fine. Turns out there are 4 closets and 1 room. All located at different parts of the entire property. Office, warehouses 1, 2, and 3. Great. Looks like Ill be walking around for awhile. So I set off. I cant get into one of the closets since my "master" key doesnt work and there is no card reader to swipe my badge with. (Ohh and I had to change my own access to my badge so I could get into these data centers to make sure they were ok. Restrict it to people that might need it but give us access to change it ourselves. Very smart. :rolleyes: ) Ok, goto closets 2-4 which take about 4-5 minutes walking to get to each one. Slowly getting further from my point of origin. Finally get to the UPS room and that is also at 93*. WTF!!!! So I call the maintenance guy back up and he is on the way.

Its 6:15 and Im still dealing with stuff due to a power outage that happened over 6 hours ago. In the end.....fuck you Com Ed!!!!
 
Old Thread: Hello . There have been no replies in this thread for 90 days.
Content in this thread may no longer be relevant. Consider starting a new thread to get fresh replies.

Thread Info