Several linux questions, with a new one

[Crusty]

So I got my new drives in the fileserver and decided against putting them in RAID 1.

When I try copying a file from the software raid array which is on my onboard IDE to my new drives on a promise ata card cpu usage skyrockets to 100%

I just upgraded my kernel to 2.6.11, there are errors in dmesg... but I don't think they mean anything

devfs_mk_dev: could not append to parent for md/0
md: raidstart(pid 1201) used deprecated START_ARRAY ioctl. This will not be supported beyond 2.6
md: could not open unknown-block(33,1).
md: autostart failed!
md: raidstart(pid 1201) used deprecated START_ARRAY ioctl. This will not be supported beyond 2.6
md: could not open unknown-block(34,1).
md: autostart failed!

but...

md: raidstart(pid 2483) used deprecated START_ARRAY ioctl. This will not be supported beyond 2.6
md: autorun ...
md: considering hdc1 ...
md: adding hdc1 ...
md: adding hda1 ...
md: created md0
md: bind<hda1>
md: bind<hdc1>
md: running: <hdc1><hda1>
raid1: raid set md0 active with 2 out of 2 mirrors
md: ... autorun DONE.

And the array comes up and works just fine. If I try and copy from the array across the network CPU usage is low, smbd only hits 15% at the highest. I've tried a different promise card, with the same results.

When I use mv, mv takes up 100% cpu...
When I use cp, kjournald takes up 90% and cp takes up 10%

Any ideas?



Old questions below
-------------------------------------------


I've finally filled up my fileserver. It currently has 2x100gb drives in RAID 1. Boot is off a scsi drive.

I want to add more drives to the computer, would it be okay if i slapped 1 more drive on each of the onbard controllers and put those two drives into RAID 1?

I've read that having more then one drive on each channel in RAID isn't a good idea.

I really do not want to put in a IDE card because the mobo in that computer get's really picky when there is an IDE card in there because of the onboard SCSI.

How do enable jumbo frames in linux permanately? I'm aware that you have to change the MTU through ifconfig, but how do I add it to the startup scripts? Running debian btw

Currently my network tops out around 25MB/s or ~200mbps, are there any other network tweaks I can do to get better performance out of my gigabit network? I've done all the cabling myself, with CAT 5E following the T568B standard. I am going to go buy new patch cords, because the quality of those are going to be much better then the ones that I crimped. I took my time with all the inwall cabling, making sure all the connections were perfect.

All of my networking equipment is in my AV rack along with my HTPC, so I have my HTPC hooked up to both the gigabit network and my 100mbit network. The 100mbit network is used for internet and wireless etc. I seperated the two networks solely so I could use jumbo frames, and this is the only link between the two networks. I haven't touched any routes or ip tables in my HTPC though. Gigabit is on 10.x.x.x and 100mbit is on 192.x.x.x

What is the easiest way to get my fileserver on the internet through my htpc in order to get updates for the OS? I was thinking that squid might be able to do it, and I would prefer to use squid really. I need to setup a private proxy for myself to the outside anyways. My biggest question with squid is can it accept connections on one nic and "proxy" them to the other nic? Basically allow my fileserver to pull debian updates through my htpc? Or is there an easier way?


I have some mp3's on my fileserver that contain non-english characters, xmms won't play some of them and i'm guessing it's because it doesn't understand some of the characters in the filename/id3 tag. What can I do to remedy this? I've allready tried installing xfont-intl-phonetic and xfree-nonfree-fonts.... or something like that.

Thx for you help

 

[drag]

Originally posted by: MCrusty
I've finally filled up my fileserver. It currently has 2x100gb drives in RAID 1. Boot is off a scsi drive.

I want to add more drives to the computer, would it be okay if i slapped 1 more drive on each of the onbard controllers and put those two drives into RAID 1?

I've read that having more then one drive on each channel in RAID isn't a good idea.

It's possible. Your performance will go down the crapper though. This is because in Raid 1 you have to write once to one drive and another time to the other drive. One PATA channel can only communicate to one drive at a time, so everytime you make a write to your drive it will take twice as long.. more or less.

I don't know all the details.. but it will suck. For the same reasons you don't want to be writing out a DVD from a drive on the same channel.

I really do not want to put in a IDE card because the mobo in that computer get's really picky when there is an IDE card in there because of the onboard SCSI.

How do enable jumbo frames in linux permanately? I'm aware that you have to change the MTU through ifconfig, but how do I add it to the startup scripts? Running debian btw

In Debian your network configurations are stored in /etc/network/interfaces text file. If your using a static network config then you just stick a 'mtu 8000" or whatever line underneath the paticular config.

See 'man interfaces'.

Test to see if it works by going:
ifdown eth0
ifup eth0
ifconfig eth0

Currently my network tops out around 25MB/s or ~200mbps, are there any other network tweaks I can do to get better performance out of my gigabit network? I've done all the cabling myself, with CAT 5E following the T568B standard. I am going to go buy new patch cords, because the quality of those are going to be much better then the ones that I crimped. I took my time with all the inwall cabling, making sure all the connections were perfect.

Ya this is probably going to be more of a function of your network interfaces and paticular configurations then the actual network if your cabling is nice.

Make sure that your using a nice switch that supports jumbo frames.. (unless your just going computer-to-computer without a switch or anything like that) Jumbo frames may not help a whole lot out with speed, but it should reduce the cpu load.. If your using a complex file transfer setup like thru SAMBA then there is a lot of tweaking you can do to increase performance.

Also harddrive I/O can limit performance, too. In real world conditions I wouldn't be suprised if a haddrive would be limited to 25MB/s or so even if they benchmark at 50MB/s.

There are numerious networking benchmark do-dads. Do some apt-get searches.

Some nicer Nics can do things like tcp/ip offload or something like that to gain performance. Maybe check that out for your paticular nics. You may be able to tweak them.

All of my networking equipment is in my AV rack along with my HTPC, so I have my HTPC hooked up to both the gigabit network and my 100mbit network. The 100mbit network is used for internet and wireless etc. I seperated the two networks solely so I could use jumbo frames, and this is the only link between the two networks. I haven't touched any routes or ip tables in my HTPC though. Gigabit is on 10.x.x.x and 100mbit is on 192.x.x.x

What is the easiest way to get my fileserver on the internet through my htpc in order to get updates for the OS? I was thinking that squid might be able to do it, and I would prefer to use squid really. I need to setup a private proxy for myself to the outside anyways. My biggest question with squid is can it accept connections on one nic and "proxy" them to the other nic? Basically allow my fileserver to pull debian updates through my htpc? Or is there an easier way?

Probably the easiest is just to setup your HTPC as a little masquarade/NAT-style router. It's pretty simple.
http://www.linuxdownloads.org/howto/Masquerading-Simple-HOWTO.html

That way your file server can initiate any internet connection it feels like, and you can port forward the paticular file services you want your external network to be able to access.

I like to setup a little local rc file, like what is commonly used on Redhat-style distros, and stick the commands that I would normally have to type out manually to get the proper masq config.
http://www.plope.com/Members/chrism/debian_rc_local_equiv

I have some mp3's on my fileserver that contain non-english characters, xmms won't play some of them and i'm guessing it's because it doesn't understand some of the characters in the filename/id3 tag. What can I do to remedy this? I've allready tried installing xfont-intl-phonetic and xfree-nonfree-fonts.... or something like that.

Ya I don't know. Maybe Xmms doesn't support UTF-8 (?) style characters, just ascii stuff. If you can write a script to get rid of those characters it may help.. Or contact XMMS people and ask them.. or just find a different mp3 player.

What I like to use is something called mpd. Or just music player deamon. It's a daemon that can be started by the system as root, or by a user and it plays songs from m3u lists. It takes a bit to setup, I use a script to generate a m3u list file for it (and I have to eliminate any unusual characters from the file/directories names). But it's nice because you don't need a app open to play music.

It has clients you can setup to control the music. You have mpc command line client, and gmpc for gnome desktop as well as some webbased clients.

I found somethings interesting lately. Found 'powertweak' which is aviable thru apt-get and this:
http://www.togaware.com/linux/survivor/index.shtml which is the Debian GNU/Linux desktop survival guide.

 

[Nothinman]

I want to add more drives to the computer, would it be okay if i slapped 1 more drive on each of the onbard controllers and put those two drives into RAID 1?

If you're doing RAID 1 in firmware it should be alright, you'll have to update fstab though. If you're using linux software RAID you'll have some work to do if you want to convert to RAID.

I've read that having more then one drive on each channel in RAID isn't a good idea.

Most ATA devices don't do disconnect or TCQ so if one drive is busy it'll monopolize the channel, so if you have 2 drives on the same channel both being used a lot you'll run into contention.

How do enable jumbo frames in linux permanately? I'm aware that you have to change the MTU through ifconfig, but how do I add it to the startup scripts? Running debian btw

You can specify the MTU size in /etc/network/interfaces.

xmms won't play some of them and i'm guessing it's because it doesn't understand some of the characters in the filename/id3 tag. What can I do to remedy this? I've allready tried installing xfont-intl-phonetic and xfree-nonfree-fonts.... or something like that.

XMMS probably doesn't do UTF-8, BMP is a GTK-2 port of XMMS so it might work better. Otherwise you'll probably have to find another app to play the music.

 

[Crusty]

Oh, i think you both kind of misunderstood my question about the raid... it was badly worded on my part.

Right now I have 2 drives /dev/hda and /dev/hdc in raid 1
I want to add 2 more drives and put them in raid 1.. so /dev/hdb and /dev/hdd

So now I will have seperate raid arrays, will that still cause problems. Generally I won't be reading/writing to both arrays at once, since this is mostly storing media files.



All the nics on the gigabit network are Intel 1000/Pro cards I believe. So I'll look around on google to see what I can turn up. They are all connected to a SMC switch which supports jumbo frames up to 9000k.

Thanks for that link about nat drag, that worked perfectly.

As for the music, bmp didn't play them and told me that it was because of UTF-8 problems... i'll talk to the xmms people and see what they say. I would really like to not have to change the files...

edit: It seems they are adding UTF-8 and UTF-16 support in the next version.... I guess I can live without these few songs until then.

Thanks for the help so far

 

[Crusty]

Oh, and is samba my only option if I want share the files with a windows box?

 

[drag]

Originally posted by: MCrusty
Oh, and is samba my only option if I want share the files with a windows box?

Nope. But it's about the best one.

 

[drag]

Originally posted by: MCrusty
All the nics on the gigabit network are Intel 1000/Pro cards I believe. So I'll look around on google to see what I can turn up. They are all connected to a SMC switch which supports jumbo frames up to 9000k.

I have a almost identical setup. The cheaper versions of the Intel 1000 (nicer ones support offloading some of the load, but that's not the model I have.). And a 5 port SMC switch.

Very nice.

Although I did some benchtesting and I don't think that my SMC can do the full 9000k jumbo frames.. it begins to run out of cache or memory or something like that. I've found that 8000 works better. They claim it does, but I think they are stretching it a bit. Very nice switch.

 

[Crusty]

I just tested with netperf and this is what I got...

htpc:/av# netperf -H 10.0.0.3
TCP STREAM TEST to 10.0.0.3
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 16384 10.00 540.36


Is that last number in mbits?

 

[lansalot]

Right now I have 2 drives /dev/hda and /dev/hdc in raid 1
I want to add 2 more drives and put them in raid 1.. so /dev/hdb and /dev/hdd

So now I will have seperate raid arrays, will that still cause problems. Generally I won't be reading/writing to both arrays at once, since this is mostly storing media files.

That will be fine. As you say, if you won't be reading/writing to both at the same time (and you're willing to suffer the performance hit when they do collide), then go for it.

 

[Crusty]

*bump* for new question

 

[Crusty]

anyone?

 

[n0cmonkey]

Which question? The netperf one or the kjournald one?

 

[Crusty]

Originally posted by: n0cmonkey
Which question? The netperf one or the kjournald one?

kjournald

I figured out the netperf one . Just had to dig around the man pages for it.

 

[n0cmonkey]

High cpu with kjournald seems to be fairly normal.

Are you using ext3? What kind of journalling is it using?

 

[Crusty]

Yeah, it's using ext3... the journalling is whatever the default is. order?

I can check in a few minutes, i've been rearranging my network gear and computers so it's now on the network atm so i can't ssh into it.

 

[Crusty]

kjournald starting. Commit interval 5 seconds
EXT3 FS on hde1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.