anyone else work at a data center?

Toggle sidebar Toggle sidebar
N

nageov3t

Lifer
Feb 18, 2004
42,808
83
91
how often does your data center have down servers for purely stupid reasons, like poorly headed cables slipping out of the port or a clumsy tech working in a rack?

my DC can't be the only one. haha. seems to happen at least once or twice a week, which I guess isn't terribly bad out of the 3000 servers at my location, though we get inventive with making up excuses for clients... mostly the results of our near-worthless night guys taking shortcuts like using an already-made cable with a broken head instead of taking 30 seconds to re-head it.
 
D

d33pt

Diamond Member
Jan 12, 2001
5,654
1
81
today i had a server go down because a UPS died. i think that was pretty stupid.
 
J

JasonCoder

Golden Member
Feb 23, 2005
1,893
1
81
Originally posted by: d33pt
today i had a server go down because a UPS died. i think that was pretty stupid.
Click to expand...

So the UPS died at the same time that you had a power outage? That's uncool is what that is.
 
B

biggestmuff

Diamond Member
Mar 20, 2001
8,201
2
0
Originally posted by: loki8481
how often does your data center have down servers for purely stupid reasons, like poorly headed cables slipping out of the port or a clumsy tech working in a rack?

my DC can't be the only one. haha. seems to happen at least once or twice a week, which I guess isn't terribly bad out of the 3000 servers at my location, though we get inventive with making up excuses for clients... mostly the results of our near-worthless night guys taking shortcuts like using an already-made cable with a broken head instead of taking 30 seconds to re-head it.
Click to expand...

:confused:
 
Muadib

Muadib

Lifer
May 30, 2000
18,110
908
126
Originally posted by: biggestmuff
Originally posted by: loki8481
how often does your data center have down servers for purely stupid reasons, like poorly headed cables slipping out of the port or a clumsy tech working in a rack?

my DC can't be the only one. haha. seems to happen at least once or twice a week, which I guess isn't terribly bad out of the 3000 servers at my location, though we get inventive with making up excuses for clients... mostly the results of our near-worthless night guys taking shortcuts like using an already-made cable with a broken head instead of taking 30 seconds to re-head it.
Click to expand...

:confused:
Click to expand...

I'm glad I wasn't the only one..
 
SparkyJJO

SparkyJJO

Lifer
May 16, 2002
13,357
7
81
We don't usually have that issue. We are really particular about our cabling, and clumsy techs don't last long ;)
 
M

melchoir

Senior member
Nov 3, 2002
761
1
0
In our primary location, almost never. In our secondary location where we don't have any "smart hands", it's happened a few times. You can give people explicit, detailed, and exact instructions, and they can still manage to F something up.
 
J

JDMnAR1

Lifer
May 12, 2003
11,984
1
0
We have a couple of older racks without integrated PDUs where power is provided by power strips zip tied to the rack itself. Given the configuration, this means that the rocker switches on said power strips are relatively easy to hit when recabling/moving/otherwise working with servers in the racks. We have had an occasion or two where lazy techs didn't split their power sources for servers (i.e. plug each power supply into different power strips) and when a power strip got turned off inadvertantly it took down the server because the redundant power wasn't so redundant after all.
 
Y

yoda291

Diamond Member
Aug 11, 2001
5,079
0
0
Originally posted by: loki8481
how often does your data center have down servers for purely stupid reasons, like poorly headed cables slipping out of the port or a clumsy tech working in a rack?

my DC can't be the only one. haha. seems to happen at least once or twice a week, which I guess isn't terribly bad out of the 3000 servers at my location, though we get inventive with making up excuses for clients... mostly the results of our near-worthless night guys taking shortcuts like using an already-made cable with a broken head instead of taking 30 seconds to re-head it.
Click to expand...

well, there's part of your problem,

you should be tossing that cable and getting a new one, not re-crimping the old one.

if you have people of varying competencies, you need to make things as boneheadedly idiot-proof as possible.
 
K

kedlav

Senior member
Aug 2, 2006
632
0
0
Originally posted by: yoda291
Originally posted by: loki8481
how often does your data center have down servers for purely stupid reasons, like poorly headed cables slipping out of the port or a clumsy tech working in a rack?

my DC can't be the only one. haha. seems to happen at least once or twice a week, which I guess isn't terribly bad out of the 3000 servers at my location, though we get inventive with making up excuses for clients... mostly the results of our near-worthless night guys taking shortcuts like using an already-made cable with a broken head instead of taking 30 seconds to re-head it.
Click to expand...

well, there's part of your problem,

you should be tossing that cable and getting a new one, not re-crimping the old one.

if you have people of varying competencies, you need to make things as boneheadedly idiot-proof as possible.
Click to expand...

Need manager-proof environment as well
 
J

JJChicken

Diamond Member
Apr 9, 2007
6,165
16
81
Originally posted by: loki8481
how often does your data center have down servers for purely stupid reasons, like poorly headed cables slipping out of the port or a clumsy tech working in a rack?

my DC can't be the only one. haha. seems to happen at least once or twice a week, which I guess isn't terribly bad out of the 3000 servers at my location, though we get inventive with making up excuses for clients... mostly the results of our near-worthless night guys taking shortcuts like using an already-made cable with a broken head instead of taking 30 seconds to re-head it.
Click to expand...

I work at a school. We had one of our hard drives in our main data server to start getting corrupted and it brought the whole network down :( Took us two weeks to fix the problem (it was working intermittendly) :laugh: Lucky I was on leave during the period and rejoined when it wass nice and dandy again :grin;
 
M

Mr Pickles

Diamond Member
Feb 24, 2006
4,103
1
0
The "datacenter" for the financial firm that I work for is located here. It hosts more then 3\4 of the servers the business runs nationwide, roughly 175 servers. Nothing serious, but they we're an HP Server shop using Cisco switching and routing. Every once and a while a drive will fail or a UPS will black out but we have the redundancy here, but never any major issues unless there is a ripple upstream or if our ISP is on the fritz. But even then we have a redundant ISP, naturally. The issues are with the smaller servers in all of the other offices that are too small to employ a technician of some sort. We are alerted when devices fail or are unavailable, but certain downtown can be avoided, or at least reduced, if we had someone there to be our "hands".

We cable our DC very well, not pretty, just very well. No hand made cables or servers sitting on other servers, bla bla bla. We've got hot and cold rows just like we should and a huge AC unit keeping the room 65 the whole time. We're pretty proud of the place.

Oh well, job security I guess...
 
ScottFern

ScottFern

Diamond Member
Oct 23, 2002
3,629
2
76
I have been contemplating whether it would be more fun and interesting to work at a data center versus working for an IT department at a small/medium size business. Which I am currently doing the latter now...
 
Brovane

Brovane

Diamond Member
Dec 18, 2001
6,140
2,413
136
I work at a remote office for a fairly large company. The Data Center at the remote office we have around 70 physical IT servers in the Data Center and around 20 non IT servers in the DataCenter.

As the senior LAN Admin I am fairly strict about cabling in the Data Center and I get after people for crappy cabling jobs on the IT side. We have never had a issue with bad cabling causing a outage.

The Data Center is powered by a 100Kva PowerWare UPS and we have 3 AC units in the DataCenter with only 2 needed to properly cool the datacenter. The DataCenter UPS is backed up by a Generator that is ran on a monthly basis with 96-hours of fuel kept onsite. The Generator can power both the DataCenter and the AC units in the DataCenter. The PowerWare UPS is maintained by a outside vendor. We have never had a server outage caused by loss of power.

For the IT rack areas we have either 2 our 4 APC rackmount PDU units at the bottom of each rack that have use the lock in connectors for the power plug. We have 2 UPS PDU units in the DataCenter so we have A and B power so 1 PDU is in B and the other is in the A side. You then plug your dual PS server into the A and B PDU units. We label everything so you can easily see it. Each rack also has a KVM unit. Power goes down under the floor and network connections go up. At the top of each rack there is a 24-port patch panel with some racks having 48-ports. We also have a fiber tray above this so all fiber goes up in the fiber tray for SAN connected equipment.

The National Corporate Data Center is a Tier IV Data Center is redundant everything and is 100,000+ square feet of raised floor. I have been to this data center and it is immaculate. The Data Center Manager is even stricter than me. He calculates the amount of BTU's of heat that a server puts out and matches it up with the BTU's of cooling he has in a area. One of the key ways to keep a data center clean (IMHO) is to not allow the IT Server/Network admins into the datacenter. The national data center has a team of people that it is there job to maintain the data center and do all cabling work in the data center. The IT people at corporate do there work on the equipment inside remotely.
 
O

OdiN

Banned
Mar 1, 2000
16,430
3
0
Well no data center here, but my servers have not been offline in the time that I've been here.

We have a big generator that kicks in if the power goes out and lots of UPS's. We've had some of the desktops out on the floor go out/reboot during a power outage, but the servers have stayed up.
 
Kaido

Kaido

Elite Member & Kitchen Overlord
Feb 14, 2004
50,440
6,578
136
No, but I did stay at a Holiday Inn last night! :D
 
S

spidey07

No Lifer
Aug 4, 2000
65,469
5
76
You need to fire everybody including the data center manager if they are hand terminating cables. Extremely bad practice, and now you see why.

If you do things "RIGHT", this kind of stuff never happens. And why do you only have a single network connection anyway? All servers are to have 4 connections - one primary, one secondary, one for out of band management and one for remote support board.
 
J

JDMnAR1

Lifer
May 12, 2003
11,984
1
0
Originally posted by: spidey07
If you do things "RIGHT", this kind of stuff never happens. And why do you only have a single network connection anyway? All servers are to have 4 connections - one primary, one secondary, one for out of band management and one for remote support board.
Click to expand...

Oh pshaw!! That is just crazy talk...at least if you work here. The only time we get to actually use both onboard NICs on our servers is if they are either cluster or load-balanced nodes. For some reason the network engineers don't like having unused ports sitting idle - something about paying for capacity you won't EVER use. :roll:
 
N

nageov3t

Lifer
Feb 18, 2004
42,808
83
91
Originally posted by: spidey07
You need to fire everybody including the data center manager if they are hand terminating cables. Extremely bad practice, and now you see why.
Click to expand...

how else would you terminate cables? I've been making them by hand for 5+ years and never had an issue.

If you do things "RIGHT", this kind of stuff never happens. And why do you only have a single network connection anyway? All servers are to have 4 connections - one primary, one secondary, one for out of band management and one for remote support board.
Click to expand...

we do what our clients pay for... if they're not paying for fail-over ethernet connections, teaming, etc, it's not hooked up.

I can't imagine any DC with 3000+ servers giving 4+ connections to each one.
 
S

Steve

Lifer
May 2, 2004
15,945
11
81
You must log in or register to reply here.

TRENDING THREADS

COMPANY
RESOURCES
FOLLOW
Top Bottom