Now to wait nervously as this rebuilds

[Red Squirrel]

Added 2 drives to my raid 5 array. going from 6 1TB drives to 8 1TB drives. Rebuilding now. D: This many drives in raid 5 make me a little nervous. My kernel is too old to grow to raid 6. I'm sure I'll be fine, but still nerve wracking, especially with my luck.

[root@borg temp]# mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Sat Sep 20 02:15:28 2008
     Raid Level : raid5
     Array Size : 4883799680 (4657.55 GiB 5001.01 GB)
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 6
  Total Devices : 8
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Jan  3 21:13:05 2013
          State : clean
 Active Devices : 6
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 11f961e7:0e37ba39:2c8a1552:76dd72ee
         Events : 0.1440052

    Number   Major   Minor   RaidDevice State
       0       8       96        0      active sync   /dev/sdg
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       3       8       48        3      active sync   /dev/sdd
       4       8      112        4      active sync   /dev/sdh
       5       8       80        5      active sync   /dev/sdf

       6       8      128        -      spare   /dev/sdi
       7       8      160        -      spare   /dev/sdk
[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# df -hl
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             433G   12G  400G   3% /
/dev/sda1             190M   25M  156M  14% /boot
tmpfs                 3.8G   48K  3.8G   1% /dev/shm
/dev/md0              4.5T  3.3T  995G  78% /raid1
[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# cat simplebenchmark.sh 
echo
dd if=/dev/zero of=test.bin bs=5000000 count=1000
echo
echo
dd of=/dev/null if=test.bin bs=5000000
echo

[root@borg temp]# ./simplebenchmark.sh 

1000+0 records in
1000+0 records out
5000000000 bytes (5.0 GB) copied, 24.2193 s, 206 MB/s


1000+0 records in
1000+0 records out
5000000000 bytes (5.0 GB) copied, 3.03028 s, 1.7 GB/s

[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# mdadm --grow /dev/md0 --raid-devices=8
mdadm: Need to backup 1344K of critical section..
mdadm: ... critical section passed.
[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# 
[root@borg temp]# mdadm --detail /dev/md0
/dev/md0:
        Version : 0.91
  Creation Time : Sat Sep 20 02:15:28 2008
     Raid Level : raid5
     Array Size : 4883799680 (4657.55 GiB 5001.01 GB)
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Jan  3 21:16:08 2013
          State : clean, recovering
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

 Reshape Status : 0% complete
  Delta Devices : 2, (6->8)

           UUID : 11f961e7:0e37ba39:2c8a1552:76dd72ee
         Events : 0.1440210

    Number   Major   Minor   RaidDevice State
       0       8       96        0      active sync   /dev/sdg
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       3       8       48        3      active sync   /dev/sdd
       4       8      112        4      active sync   /dev/sdh
       5       8       80        5      active sync   /dev/sdf
       6       8      128        6      active sync   /dev/sdi
       7       8      160        7      active sync   /dev/sdk
[root@borg temp]#

Will be interesting to see the IO speed after this. I wonder if I've already reached a plateau, or if I can actually get more out of it. I'm mostly doing this for disk space and not performance mind you, but always nice to get some extra performance too.

There's also a hot spare that's not added but when I was doing testing it had lot of smart errors so think I'll have to RMA it. Using all WD Blacks 1TB. I missed out on the cheap 3TB drives pre flood, but at slightly under 100 bucks these 1TB were hard to resist. NCIX had a sale. All my slots are full now in my server. This will give me about 1.7TB of extra space. By the time I remotely come close to using all that I'll probably have built a SAN anyway. I rarely delete data, I just add more disk space. :awe:

 

[xanis]

Holy Christ, 8TB? Do you have the world's largest porn collection?

 

[Red Squirrel]

Believe it or not, there's no porn. Lot of the disk space is VMs though, I always preallocate, I've heard it has better performance, and whether or not that's true, at least it lets me reserve that space so it's dedicated to the VM. Then I have lot of random crap like documents, code, backups of code, backup rotations of certain more important code (so I can go back to a specific time), web site stuff, more backups, and pretty much every type of file you can think of from movies, music, pictures etc... It's my "cloud". I actually don't anticipate it growing all that much more TBH, but it's nice to have that extra buffer.

Also helps I don't get stuck fixing people's PCs as much. I also keep images of any PCs I have to rebuild that way next time they come see me I can just restore the image.


2% done! :awe: Turned off all VMs to speed it up though.

 

[rudeguy]

why are you posting this in OT?

 

[datalink7]

Believe it or not, there's no porn.

Not.

 

[dud]

why are you posting this in OT?

This is Red Squirrel you're asking. You realize that, don't you?

 

[Dirigible]

Holy Christ, 8TB? Do you have the world's largest porn collection?

8tb is a lot?


*Dirigible hides his server. Then looks at it and faps. Then hides it again.

 

[Red Squirrel]

It's going to be more like 6.3TB too. Damn hard drive manufacturers and their false numbering schemes. 1KB = 1024 bytes damnit!

 

[apac]

I bought 6 3TB drives a year ago when the prices were dirt cheap. $80/each if i recall.

18TB server ftw!

 

[yh125d]

I had 11TB worth of JBOD once. All porn and torrents


Whittled down to 2TB mirrored now. Only partly porn

 

[velillen]

Ive got a 19tb unraid server (well 19tb for data, its got a 3tb parity plus 1tb cache drive as well). Just added 4 3tb drives that went on sale for black friday ish time frame.

Mostly its a media server. Ripped blurays and tv shows. Plus backups of pictures, music, documents, and images of the htpc's for quick restores. No porn....that stays on the desktop.


Its creepy with media how fast you can fill it up. I started with 8tb thinking that would last a long time. Didnt even make it a year till i had to add more drives.

 

[Red Squirrel]

I bought 6 3TB drives a year ago when the prices were dirt cheap. $80/each if i recall.

18TB server ftw!

Woah sweet. Even pre flood they were still like 200 bucks here in Canada.


11% So far so good on the rebuild.

 

[Dirigible]

Ive got a 19tb unraid server (well 19tb for data, its got a 3tb parity plus 1tb cache drive as well). Just added 4 3tb drives that went on sale for black friday ish time frame.

Mostly its a media server. Ripped blurays and tv shows. Plus backups of pictures, music, documents, and images of the htpc's for quick restores. No porn....that stays on the desktop.


Its creepy with media how fast you can fill it up. I started with 8tb thinking that would last a long time. Didnt even make it a year till i had to add more drives.

That's been my experience too. Blu rays fill it up FAST. I really like unraid. Even a non-computer guy like me can make it work.

 

[RearAdmiral]

Blarg raid 5. Have you considered other types? May you have no UREs.

 

[Red Squirrel]

98% LOL

I've considered raid 6, but my kernel does not support a live rebuild. I'd have to make a whole new array then move the data over. Too much work.

Eventually I want to build a SAN, and I may look at ZFS then, or multiple smaller raid 5 volumes. Raid 10 for VMs. Though I kinda like the idea of having one giant volume. ZFS is probably better suited for that. Linux mdraid is quite solid mind you, but any type of raid 5/6 rebuild gets kinda ridiculous with this many drives.

 

[SandEagle]

why are you posting this in OT?

this is a tech forum. just because you have a basket weaving certificate does not mean the rest of us have no clue what OP is talking about.

btw, i'm kidding.

 

[Red Squirrel]

Yay all done. Ran resize2fs to resize the file system and I now have 6.3TB. Should be good for quite a long time.

[root@borg temp]# mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Sat Sep 20 02:15:28 2008
     Raid Level : raid5
     Array Size : 6837319552 (6520.58 GiB 7001.42 GB)
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 8
  Total Devices : 9
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Jan  4 17:54:24 2013
          State : clean
 Active Devices : 8
Working Devices : 9
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 11f961e7:0e37ba39:2c8a1552:76dd72ee
         Events : 0.2089486

    Number   Major   Minor   RaidDevice State
       0       8       96        0      active sync   /dev/sdg
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       3       8       48        3      active sync   /dev/sdd
       4       8      112        4      active sync   /dev/sdh
       5       8       80        5      active sync   /dev/sdf
       6       8      128        6      active sync   /dev/sdi
       7       8      160        7      active sync   /dev/sdk

       8       8      144        -      spare   /dev/sdj

[root@borg temp]# 
[root@borg temp]# df -hl
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             433G   12G  400G   3% /
/dev/sda1             190M   25M  156M  14% /boot
tmpfs                 3.8G   48K  3.8G   1% /dev/shm
/dev/md0              6.3T  3.3T  2.7T  56% /raid1

 

[bradcollins]

Having 8 drives in raid 5 would worry me too without an offline backup. I tend to buy drives of the next size up (ie going from 500 to 750 rather than 500 to 1tb) so I have fairly new drives at all times and that I can also keep the old drives in a Seagate 20 drive box as an offline copy for the array.

As you only have 1tb drives, not 3tb drives, I wonder if selling the 1tb drives, or using them as an offline backup makes more sense and changing to 4x 3tb in raid 5. Obviously you need to make sure your motherboard/raid card and kernel support 3tb drives, but it makes much more sense to me having fewer larger drives. I'm unsure what the second hand market is like over there for drives, but I know over here selling 8x 1tb would allow me to buy about 3-4x 3tb drives.