Old News & A LOT of bitching!
I'd actually wanted to post about this a while ago. Quite a while, in fact. But I'm lazy, and haven't been keeping up with the blog the way I had been. Shame on me.
Part of the reason I haven't posted in a while is my host moved the server last week, so stuff has been down. Should be up by now. If you're reading this, I would assume all is back to normal.
The Old News
The first blurb is about the IT department at work. My buddy took his laptop in because he can't get on the wireless network. The problem is that the WiFi is WPA protected, and no one knows the keys except IT. The IT guy, John Gilis (sp?)--some 40+ guy with long curly black hair (spooky) messes with it for about 5 minutes. Says it should be connecting (the network shows up in the available list) but it won't connect because my buddy had installed Yahoo! Messenger. His actual words were "You have unauthorized software on your corporate machine. It won't connect because Yahoo! is an unauthorized program. And its sucking the wireless signal from your card & preventing you from getting on the network."
So, either this guy is a dick, or he's the biggest moron ever.
What's with people turning over half finished crap as being done? Saturday night, there was an upgrade that was supposedly completed on the day shift. When I got there, the machine was still torn apart. I took a look around, and didn't really think that anything had been started, much less finished. After about an hour of getting the run around from my own shift supervisiors, someone else finally realized that the work wasn't done, or if it was, it wasn't right.
6 hours later, when it was finally finsihed, it was obvious that the people who'd supposedly completed the upgrade had gotten everything installed, but did such a great job that none of it was installed properly and none of it worked.
Not only that, but when the day shift supervisor got in Sunday morning, he started bitching about work that we hadn't done that night. Stuff that he'd emailed ONE friggin guy about (not the lead or supervisor, another tech, and NOT the guy actually working on the machine--that was me)! First off, he left us a shitty machine that wasn't ready (like he'd told our supers it was) and didn't leave complete instructions. And he was mad at us about it.
Then, Sunday night, the first things in the passdown notes were complaints about how our shift turned over the machine with some busted cables & a pump problem. That really pissed me off, because the day shift replaced that pump the day before, as part of the upgrade, and had obviously fucked that up too, but were trying to blame us for not fixing it for them. On top of that, they once again walked away in the middle of a procedure with very little notes as to what they'd done, & what needed to be done
More work bitching. LN was down this weekend. No one could find a hard copy of the procedures for test. So we were pretty much flying blind the last half shift Saturday night/Sunday morning & again all night Sunday.
The confusion was only complicated more by the poor passdown of information regarding some specialized testing going on. Not that the tests were that special or difficult or whatever, but that it was different from the norm. And very poorly documented.
It wouldn't be so bad if this was a once in a while occurance, but its rampant. The technical doucmentation is very lacking, all across the board. We're constantly using inconsistent procedures with out of date and/or just plain inacurate schematics.
There's no easy way to track common problems & their solutions. When I suggested one several months ago, I was laughed off by my supervisors boss & told that "we don't need such a thing." There's no record of normal values. No "normal operating ranges" beyond personal experiance. Now, thats fine for the poor bastards that have been in Final Test for 20 years, but I'm pretty much shit out of luck.
But, I digress. The issue was that one system went down, and EVERYONE, presumably world wide, was prevented from accessing the maintenance & assembly procedures, logs, etc for EVERY machine. And no one could come up with paper copies.
One question is why there's only one system. Why aren't there redundant backups of that system?
I also want to know why they use LN for this system. A cluster of webservers could host the procedures & logs as HTML pages or a PHP/CF/ASP application. This would enable several servers worldwide to split the dataload. Not to mention, provide a backup.
No matter the case, if it was a HD problem, they should've had an immediate solution for replacing or restoring the bad drive/drives & moving on. If it was a system problem, shut the damn thing off, pull the drives out, and put them in a machine that works... Not that there aren't litteraly tons of DELL Inspiron (or whatever) servers in that building. I mean, come on, we were winging it for over 24 hours. Not safe when you're working on multimillion dollar machines with High Voltage, toxic chemicals, and lots and lots of moving parts.