Ext2 vs Ext3 vs JFS vs ReiserFS vs XFS.. large file speed

[drag]

I wanted to find out which format was the speediest for large file support. So I found a benchmarking utility called Bonnie++ and ran some tests on my new harddrive. A Western Digital 120gig 7200 rpm special edition with the 8meg cache.

I figured that I might as well document the tests and post it here so that anyone who cares can see the results.

hdparm -i /dev/hda

Model=WDC WD1200JB-00DUA3, FwRev=75.13B75, SerialNo=WD-WMACM1759228
Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
RawCHS=16383/16/63, TrkSize=57600, SectSize=600, ECCbytes=74
BuffType=DualPortCache, BuffSize=8192kB, MaxMultSect=16, MultSect=16
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=234441648
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5
AdvancedPM=no WriteCache=enabled
Drive conforms to: device does not report version: 1 2 3 4 5 6

hdparm /dev/hda

/dev/hda:
multcount = 16 (on)
IO_support = 1 (32-bit)
unmaskirq = 1 (on)
using_dma = 1 (on)
keepsettings = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 14593/255/63, sectors = 234441648, start = 0

hdparm -tT /dev/hda

/dev/hda:
Timing buffer-cache reads: 128 MB in 0.33 seconds =387.88 MB/sec
Timing buffered disk reads: 64 MB in 1.36 seconds = 47.06 MB/sec


The rest of the system is a 1700+ amd thunderbird overclocked to 1837mhz running on a nforce2 motherboard at a 166mhz bus. 512Megs of ram, but with 32 megs shared with the on-board video.

The OS is Slackware 9.0 and using tools that came with the distro exept for the bonnie benchmark. The kernel is 2.4.22 vanilla that I reconfigured to fit my hardware. The xfs patch was applied to it, but that was after all the other tests were run.

bonnie++ benchmarking 10 3g files (actually three 1 gig files put together each)

regular ext3
Version 1.03_______------Sequential Output------ --Sequential Input- --Random-
____________________-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine________Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
marvin___________3G 20581 98 46242 23 20777 7 28266 95 44182 8 112.0 0
____________________------Sequential Create------ --------Random Create--------
____________________-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
______________files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
_________________10 3158 99 +++++ +++ +++++ +++ 2951 99 +++++ +++ 10545 99

ext3 mounted -o noatime -o data=writeback

Version 1.03_______------Sequential Output------ --Sequential Input- --Random-
____________________-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine________Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
marvin___________3G 20504 96 48124 22 20888 6 28350 93 44272 7 117.0 0
____________________------Sequential Create------ --------Random Create--------
____________________-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
______________files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
_________________10 3210 98 +++++ +++ +++++ +++ 2521 99 +++++ +++ 11203 99

reiserfs mounted -o notail

Version 1.03_______------Sequential Output------ --Sequential Input- --Random-
____________________-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine________Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
marvin___________3G 20221 96 46831 22 20766 7 27814 93 43839 9 114.6 0
____________________------Sequential Create------ --------Random Create--------
____________________-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
______________files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
_________________10 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

Jfs

Version 1.03_______------Sequential Output------ --Sequential Input- --Random-
____________________-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine________Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
marvin___________3G 14791 66 44636 11 21526 5 28743 96 48847 8 114.6 0
____________________------Sequential Create------ --------Random Create--------
____________________-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
______________files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
_________________10 8098 25 +++++ +++ 4606 9 5421 40 +++++ +++ 4440 19

ext2

Version 1.03_______------Sequential Output------ --Sequential Input- --Random-
____________________-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine________Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
marvin___________3G 21526 97 50184 10 21193 5 28225 94 44267 7 116.1 0
____________________------Sequential Create------ --------Random Create--------
____________________-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
______________files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
_________________10 6810 100 +++++ +++ +++++ +++ 6951 99 +++++ +++ 16839 98

xfs

Version 1.03_______------Sequential Output------ --Sequential Input- --Random-
____________________-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine________Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
marvin___________3G 24898 98 49965 11 22281 6 28762 96 48878 8 117.7 0
____________________------Sequential Create------ --------Random Create--------
____________________-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
______________files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
_________________10 3134 20 +++++ +++ 2905 21 3081 23 +++++ +++ 1369 10



Looks to me that XFS is the winner here in terms of performance. Which is cool since I like xfs. I guess this shouldn't be to suprising since it was orginally designed by SGI for their Unix workstations, which are commonally used in high-end graphics and digital rendering/editing.

One thing that is interesting is that the high cpu utilization when writing certain types of files to the IDE harddrive.

I ran the postmark benchmark, too. but that was unable to deal with files much larger then 100 megs, so it wasn't much use to me. In that test ext2 came out on top for small file handling speed. However I doubt I would use it since even though the journalling adds overhead the decrease in performance is very slight and the benifits far out way the slight difference in performance.

 

[Electrode]

XFS is indeed fast, but it doesn't seem to handle crashes very well. Only ext2/3 is worse.

 

[drag]

That's funny, because reiserfs was the only one that destroyed a partiton on a power outage. XFS I haven't had a problems with, and ext3 once fubar'd a /etc/ partition after a botched overclocking attempt on a different computer, however the information was retreivable.

But that's from my own limited experiances.

 

[xcript]

Originally posted by: drag
Looks to me that XFS is the winner here in terms of performance. Which is cool since I like xfs.

Interesting results, I'll be sure to bring them to the attention of all my ReiserFS touting buddies.

I've been quite happy with the performance myself. Only had a few minor problems after hardlocks, etc. but nothing I couldn't fix.

Here's to XFS. :beer:

 

[NuclearFusi0n]

Originally posted by: xcript

Originally posted by: drag
Looks to me that XFS is the winner here in terms of performance. Which is cool since I like xfs.

Interesting results, I'll be sure to bring them to the attention of all my ReiserFS touting buddies.

Try it with small files.

Reiser4 will make this all irrelevant though - nothing will compare!

 

[jhu]

how good is jfs compared with the other fs in terms of reliability? i was thinking of using jfs the next time i install linux on something

 

[drag]

Originally posted by: NuclearFusi0n

Originally posted by: xcript

Originally posted by: drag
Looks to me that XFS is the winner here in terms of performance. Which is cool since I like xfs.

Interesting results, I'll be sure to bring them to the attention of all my ReiserFS touting buddies.

Try it with small files.

Reiser4 will make this all irrelevant though - nothing will compare!

Hopefully reiser will rock.

I did this because I want to use MythTV as a PVR.. So I will be watching files being encoded, recorded to the disk, then read from the disk, decoded, and then put on the screen. I needed to find out which would work for the massive files that will be generated...

Here's the results for a earlier test that couldn't be used since it was all about small files.
This was using postmark test that was designed for file servers like e-mail or document servers with gobs of small files. These were the largest files it could handle. The set size is in bytes so the largest files were were only about 100 megs. Any larger and it would crash the program. If you set the number of transactions up to about 50 or so it would even out the bytes writen to bytes read ratio, but it would crash the program in ext2. Probably again to the file sizes much larger then the program was designed to imitate.

pm>set size 50000000 100000000
pm>set number 10
pm>set transactions 20
pm>set read 50000000
pm>set write 50000000


ReiserFS

Time:
68 seconds total
52 seconds of transactions (0 per second)

Files:
19 created (0 per second)
Creation alone: 10 files (0 per second)
Mixed with transactions: 9 files (0 per second)
11 read (0 per second)
9 appended (0 per second)
19 deleted (0 per second)
Deletion alone: 8 files (8 per second)
Mixed with transactions: 11 files (0 per second)

Data:
1010.79 megabytes read (14.86 megabytes per second)
1742.41 megabytes written (25.62 megabytes per second)


ReiserFS mounted with the no-tail option

Time:
68 seconds total
52 seconds of transactions (0 per second)

Files:
19 created (0 per second)
Creation alone: 10 files (0 per second)
Mixed with transactions: 9 files (0 per second)
11 read (0 per second)
9 appended (0 per second)
19 deleted (0 per second)
Deletion alone: 8 files (8 per second)
Mixed with transactions: 11 files (0 per second)

Data:
1010.79 megabytes read (14.86 megabytes per second)
1742.41 megabytes written (25.62 megabytes per second)

ext2

Time:
65 seconds total
50 seconds of transactions (0 per second)

Files:
19 created (0 per second)
Creation alone: 10 files (0 per second)
Mixed with transactions: 9 files (0 per second)
11 read (0 per second)
9 appended (0 per second)
19 deleted (0 per second)
Deletion alone: 8 files (8 per second)
Mixed with transactions: 11 files (0 per second)

Data:
1010.79 megabytes read (15.55 megabytes per second)
1742.41 megabytes written (26.81 megabytes per second)

ext3

Time:
73 seconds total
56 seconds of transactions (0 per second)

Files:
19 created (0 per second)
Creation alone: 10 files (0 per second)
Mixed with transactions: 9 files (0 per second)
11 read (0 per second)
9 appended (0 per second)
19 deleted (0 per second)
Deletion alone: 8 files (8 per second)
Mixed with transactions: 11 files (0 per second)

Data:
1010.79 megabytes read (13.85 megabytes per second)
1742.41 megabytes written (23.87 megabytes per second)

ext3 mounted with -o noatime -o data=writeback

Time:
70 seconds total
53 seconds of transactions (0 per second)

Files:
19 created (0 per second)
Creation alone: 10 files (0 per second)
Mixed with transactions: 9 files (0 per second)
11 read (0 per second)
9 appended (0 per second)
19 deleted (0 per second)
Deletion alone: 8 files (8 per second)
Mixed with transactions: 11 files (0 per second)

Data:
1010.79 megabytes read (14.44 megabytes per second)
1742.41 megabytes written (24.89 megabytes per second)

JFS

Time:
74 seconds total
57 seconds of transactions (0 per second)

Files:
19 created (0 per second)
Creation alone: 10 files (0 per second)
Mixed with transactions: 9 files (0 per second)
11 read (0 per second)
9 appended (0 per second)
19 deleted (0 per second)
Deletion alone: 8 files (8 per second)
Mixed with transactions: 11 files (0 per second)

Data:
1010.79 megabytes read (13.66 megabytes per second)
1742.41 megabytes written (23.55 megabytes per second)


On a slower HD using Ext3...

Time:
125 seconds total
94 seconds of transactions (0 per second)

Files:
19 created (0 per second)
Creation alone: 10 files (0 per second)
Mixed with transactions: 9 files (0 per second)
11 read (0 per second)
9 appended (0 per second)
19 deleted (0 per second)

Deletion alone: 8 files (8 per second)
Mixed with transactions: 11 files (0 per second)

Data:
1010.79 megabytes read (8.09 megabytes per second)
1742.41 megabytes written (13.94 megabytes per second)

 

[drag]

Originally posted by: jhu
how good is jfs compared with the other fs in terms of reliability? i was thinking of using jfs the next time i install linux on something

I have no idea.. I don't think that people use jfs a whole lot, but since it is from IBM they tend to error (greatly) on the side of stability over performance.. so I am guessing that this would be a pretty reliable

 

[Nothinman]

XFS is indeed fast, but it doesn't seem to handle crashes very well. Only ext2/3 is worse.

I've been using XFS since the 1.0 release and it's handled all my crashes just fine. The only issue was that files could get filled with NULLs if they had dirty buffers there weren't flushed, but the data would have been lost anyway so I don't see the big deal there. Sadly I have had to convert one of my sparc64 boxes filesystems to reiserfs because XFS would oops on journal recovery. I'd like to set the fs up on a software mirror soon, maybe that'll be a time to see how jfs works on sparc64...

Similar experience with ext3, I have it on a few boxes and it has never given me a problem or lost any data for me.

Reiserfs on the other hand ate a filesystem of mine once and since they keep changing the on-disk format and rewriting things I'm going to stay very far away from it whenver possible, atleast until they decide what they want to do and settle things down.

 

[GhettoFob]

I'm planning on formatting my harddrives this weekend and I also have a MythTV setup. Which filesystem do you think is best and when is reiser 4 going to come out?

 

[BurnItDwn]

whats a crash ??? I thought that feature was only for windows???

 

[drag]

In linux land, we considure brown outs and power outages crashes.

 

[drag]

Originally posted by: GhettoFob
I'm planning on formatting my harddrives this weekend and I also have a MythTV setup. Which filesystem do you think is best and when is reiser 4 going to come out?

I have no clue. Right now it seems that xfs is the best for MythTV according to my tests... however if you look at the mailing lists their was one guy who thought that vfat was the best... soo.

Reiser4 is promising a lot in terms of performance, but pretty much any format will do, there isn't much performace difference between the different formats. It's more dependent on hardware and personal preference.

 

[BurnItDwn]

heh ... my APC UPS allows me to properly power down in the event of a power outage ... that said ... I have never had linux "crash" (unless you count X dumping as a crash ... though that does not require a reboot, and It hasn't happened in at least a year or so.)