My OCD kicked in on this new server build

[Homerboy]

Building up a SuperMicro chassis and installed 8x2TB HDDs into the 12 bay enclosure.

The bays are apparently connected to the controller card in columns. For some reason, likley due to having an even number instead of an odd number this bothered me. Not a big deal, but personally, I'd rather have them go row by row since most people are going to install an even # of drives. Tolerable.

But then I went to build the RAID10 array and noticed the slot numbers are out of order meaning they hooked up the bays out of order to the controller card.

You can see in the pic the Slot numbers read:
0
1
2
3
4
5
7
8

Yet the physical slots on the face of the chassis are populated in order.
I resisted pulling it apart and trying to reorder the cables.
But I didn't sleep well.

 

[repoman0]

I am guessing this isn't one of the 826s with the expander backplane, but with the individual breakout cables? I think I would have redone the cabling upon seeing that 6 was missing

I will say that column by column is the only natural way to do it. Row by row would be insanity.

 

[tynopik]

<delete>

 

[Exterous]

Sounds like a good receipt for hosing the array by accident and pulling the wrong drive to deal with a failure

 

[Homerboy]

I am guessing this isn't one of the 826s with the expander backplane, but with the individual breakout cables? I think I would have redone the cabling upon seeing that 6 was missing

I will say that column by column is the only natural way to do it. Row by row would be insanity.

You do row by row or column by column based on which is an EVEN number. Who installs odd numbers of drives?! (I know it happens, but it's way less likely than even # of drives)
I still have to insall the OS and get it in the rack - I still have time to adjust the cables....

 

[Homerboy]

Sounds like a good receipt for hosing the array by accident and pulling the wrong drive to deal with a failure

Possibly - but on the same note, the array will indicate what # drive and if you just count the slots...
Ugh, Now you've REALLY made me worry

 

[Exterous]

Possibly - but on the same note, the array will indicate what # drive and if you just count the slots...
Ugh, Now you've REALLY made me worry

Except for that time disk 7 fails, whoever is counting doesn't notice they shouldn't count "5...6...7..." or that they seem to be a disk short at the end. Instead they pull the wrong drive. I mean it shouldn't happen but then I also had a very large customer suddenly power off their entire datacenter because someone did something really dumb during a flywheel test. When it comes to data storage I always tend to err on the side of my OCD.

Also if you expand I'm assuming 6 will show up in the RAID list at some point meaning if you count drive bays anything after 5 is wrong?

 

[Homerboy]

Except for that time disk 7 fails, whoever is counting doesn't notice they shouldn't count "5...6...7..." or that they seem to be a disk short at the end. Instead they pull the wrong drive. I mean it shouldn't happen but then I also had a very large customer suddenly power off their entire datacenter because someone did something really dumb during a flywheel test. When it comes to data storage I always tend to err on the side of my OCD.

Also if you expand I'm assuming 6 will show up in the RAID list at some point meaning if you count drive bays anything after 5 is wrong?

Yeah. All valid points. GD it.
I'm going to have to open it up now.

 

[Exterous]

Yeah. All valid points. GD it.
I'm going to have to open it up now.

Heh - sorry

 

[Homerboy]

Heh - sorry

AND rebuild the damned array (which took overnight)
hrrppmph.

 

[Homerboy]

Sadly, I don't think there's much I can do about it. physical bays must be internally wired incorrectly. I can't access it.

 

[repoman0]

So it is one of the backplanes with real non-breakout SAS cables. Strange that it's appearing out of order. Isn't the "wiring" on the PCB of the backplane? There's really nothing to be wired incorrectly, just printed circuits.

 

[Genx87]

What am I missing? That your controller card starts its device listing at port 0 instead of 1?

Your card should also be able to blink a dead drive so you know which one to replace.

Edit: Oh I see, #6 is out of order. Oh well. Provided all slots light up dont worry about it and let your OCD go worry about something else You should be able to blink the dead drive. Or it should blink on its own when it fails.

 

[Homerboy]

Figured it out.
Bay0/Port0 is not top most left
It's bottom most left.
So looks like

2 -- 5 -- 8 -- 11
1 -- 4 -- 7 -- 10
0 -- 3 -- 6 -- 9

So, I switched HDDs in 8 and 6 around and now the Array reports in sequential order.
I've built a lot of servers - SuperMicro, Dell, HP, whatever -- I don't ever recall top to bottom drive order like that.

 

[repoman0]

Wow, I had the thought in my mind that this is how it would appear if it was upside down but didn't bother saying it because it doesn't make any damn sense.

 

[Homerboy]

Wow, I had the thought in my mind that this is how it would appear if it was upside down but didn't bother saying it because it doesn't make any damn sense.

Funny/sad thing is this going out of my hands onsite and being installed in a remote Colo. So as @Exterous pointed out, when a replacement is needed I hope the remote hands does it right
*sigh*

 

[dave_the_nerd]

Sounds like a good receipt for hosing the array by accident and pulling the wrong drive to deal with a failure

Nah. You enable the tattle light for the failed disk before you go to the rack, so you don't have to remember.

 

[Carson Dyle]

Funny/sad thing is this going out of my hands onsite and being installed in a remote Colo. So as @Exterous pointed out, when a replacement is needed I hope the remote hands does it right
*sigh*

Label the drive carriers with their bay numbers to avoid confusion.The little flat areas on the carrier latches are designed to hold labels.

On the label (better yet, a 2nd label), I would also add the last 4 or 5 digits of the drive's serial number in that carrier. If someone runs a SMART utility identifying a problem drive, the only way to identify the corresponding physical drive will be by serial number.

 

[Exterous]

Figured it out.
Bay0/Port0 is not top most left
It's bottom most left.
So looks like

2 -- 5 -- 8 -- 11
1 -- 4 -- 7 -- 10
0 -- 3 -- 6 -- 9

So, I switched HDDs in 8 and 6 around and now the Array reports in sequential order.
I've built a lot of servers - SuperMicro, Dell, HP, whatever -- I don't ever recall top to bottom drive order like that.

I've never seen that either. But at least its better than 6 being in completely the wrong spot. Maybe a good time to break out the label maker...

Nah. You enable the tattle light for the failed disk before you go to the rack, so you don't have to remember.

Sure except when the light doesn't work or where there isn't the ability to make a light blink. I've seen both cases

 

[Homerboy]

Label the drive carriers with their bay numbers to avoid confusion.The little flat areas on the carrier latches are designed to hold labels.

On the label (better yet, a 2nd label), I would also add the last 4 or 5 digits of the drive's serial number in that carrier. If someone runs a SMART utility identifying a problem drive, the only way to identify the corresponding physical drive will be by serial number.

It came with cute little stickers for the HDD trays. I will apply those.
When we take it to the colo, I'll bring it up with them and see what they think. I'm not OVERLY worried about a drive being mis-swapped. As others have pointed out there's dummy lights that will turn on/flash.
I'm just happy my I figured it out and it's only "upside down" and not a single drive out of order. That settles my OCD down and makes future mistakes less prone too I guess.

Just REALLY weird drive order IMO.

 

[Carson Dyle]

I'm not OVERLY worried about a drive being mis-swapped. As others have pointed out there's dummy lights that will turn on/flash.

Sure. When the controller drops a drive because it's failed (or the controller thinks it has). But how would you pull the correct drive if SMART were telling you that a failure was imminent? You need to know which drive is in which bay.

 

[Genx87]

Sure. When the controller drops a drive because it's failed (or the controller thinks it has). But how would you pull the correct drive if SMART were telling you that a failure was imminent? You need to know which drive is in which bay.

Most quality controllers or software allows to blink any drive regardless of its state. This has been pretty standard for quite some time. In HPs case if a drive is predicted to fail it will light the drive up automatically even if it isnt in a failed state. In 2018 if a new box doesnt have these features I would question what the hell you are putting into your data center.

 

[Genx87]

OP that is pretty funny. I didnt even think they were going from bottom to top on how they connected the drives. Now I have to go home and check out my R510 and see how they are connected. Really curious now

 

[Carson Dyle]

Most quality controllers or software allows to blink any drive regardless of its state. This has been pretty standard for quite some time. In HPs case if a drive is predicted to fail it will light the drive up automatically even if it isnt in a failed state. In 2018 if a new box doesnt have these features I would question what the hell you are putting into your data center.

So, Homer, does your software enable you to do that? Preferably from within the OS, without booting into the RAID BIOS.

 

[Homerboy]

So, Homer, does your software enable you to do that? Preferably from within the OS, without booting into the RAID BIOS.

haven't loaded an OS on it yet. But using previous SM cases/hardware I'm pretty confident that their Management software allows for it.