How to interpret /proc/meminfo

[xtknight]

cat /proc/meminfo reports that I have ~34 TB allocated. Yet, at the moment, my system seems very fast, even my VM under VMware seems fast. uptime reports: 14:30:03 up 23:01, 3 users, load average: 0.51, 0.43, 0.42

I've noticed though that every time my system slows to a halt (music pauses, mouse freezes), 34 TB has been allocated right after I start up VMware. After a while, the system stabilizes and gets fast again. What's going on here?

I'm using Ubuntu Dapper 6.06 64-bit. My friend's 32-bit Dapper system also has VMware and he doesn't have this issue. This has happened at least five times on mine. There's hardly a time I type meminfo and don't read 34 TB.

Kernel (which is custom compiled with reiser4 support): Linux ubuntu 2.6.16.18 #1 SMP PREEMPT Sun May 28 16:28:58 EDT 2006 x86_64 GNU/Linux

My friend is using a default 2.6.15-25-686 Ubuntu kernel, so maybe that is why he doesn't have the problem. Is there a bug in 2.6.16.18?

 

[Nothinman]

VmallocTotal is the total amount of virtually address space available to the kernel.
VmallocUsed is the amount that's currently used.
VmalloccChunk is the largest contiguous chunk available.

I would be really surprised if the Total value ever changed. From the attached output, you only have 304020kB vmalloc space currently used, not 34TB. That seems high, since I only have 7088kB allocated on this notebook, and even after loading the VMWare modules and resuming a VM I only have 7304kB allocated.

 

[xtknight]

You're right. The Total never did change. I must have been imagining something, or I was on a 32-bit distro.

I have another question about memory management. Over time, the Guest OS in my VMware is DOG SLOW, to the point where it takes a solid five minutes of processing time to close one process. Surely there must be something I can do to speed this up? Meminfo reports 15M of physical memory free and roughly ~2 GB of swap used (whoa). Can I prioritize which apps get physical memory and which get slow swap? There are times where I'd much rather have the 40 open Mozilla tabs swapped than the VMware that I need to use. I suppose it's just taking so long to swap it's ridiculous. I have 2 GB of total physical memory and 7 GB of swap. And phew, what is hogging 4 GB of memory?

Do I really need to insert more RAM into my PC or what? Is my swap file that slow? It just amazes me. There is no way to reliably shut down my Windows guest operating system without waiting 30 minutes (yes, literally) for it to close all the apps on its own, and it requires user intervention and attention since all of the documents haven't been saved. It really is a pain. In the past I've just killed the whole VMware process I got so tired of waiting. That's hardly ideal when I have work documents that I need (I do back them up to floppy and network). Likely the only thing I can do is get more memory. Since all my slots are full, I'll consider getting a Gigabyte i-RAM (rev 2?) and using that as my swap. Does that sound like a good idea?

My usage patterns are quite hard on this PC:

Open Linux, typically 2 browser windows with 20 tabs each. IM client, XMMS, console.

VMware: Windows: under which I run Photoshop, Dreamweaver, Opera, IE, Firefox (to test my web site). Often I have multiple browsers/tabs open under Windows as well.

All this with at least three days between reboots.

One good thing I can say is that stability has been rock solid throughout all this use, both on the guest OS, VMware itself, and GNOME/Linux. After defragmenting the "virtual disk" (VMware's hard disk), everything seems fast again, but how can it get so fragmented? I guess even though reiserfs (host file system) may not fragment a lot, Windows still has control of the virtual disk and can mangle it with NTFS fragments in any way it pleases.

The root of the problem is likely that this morning, I used Linux only (I didn't restore the VMware window). Then VMware got swapped out and hell breaks lose. My Linux itself is actually quite fast, it's just the guest OS that gets the shaft. Somehow I need to prevent it from being swapped out. I remember an option in Windows that stated that it can prioritize for the focused application or for background processes. I need to find the Linux equivalent.

BTW, here is my uptime/Meminfo in one of the "dog slow" scenarios:

 

[Nothinman]

Can I prioritize which apps get physical memory and which get slow swap?

No. The apps themselves can call mlock(), but there's no external way to set that.

Since all my slots are full, I'll consider getting a Gigabyte i-RAM (rev 2?) and using that as my swap. Does that sound like a good idea?

Not really, I've never seen VMWare perform that poorly on Linux. I have a machine with 1G memory and a XP child in VMWare with 256M allocated performs fine. At least it used to, I haven't ran it in a while.

I guess even though reiserfs (host file system) may not fragment a lot, Windows still has control of the virtual disk and can mangle it with NTFS fragments in any way it pleases.

Well the internal NTFS structures will fragment, but I don't run reiserfs on any filesystems that I actually care about so I don't know about it. Try running filefrag on the file to see how fragmented it is when it's running poor, I doubt it matters but you never know.

Somehow I need to prevent it from being swapped out.

Did you change any of the options on the memory tab of VMWare's preferences? There's one that says "Fit all virtual machine memory into reserved host RAM" which might mlock() it for you.

I remember an option in Windows that stated that it can prioritize for the focused application or for background processes. I need to find the Linux equivalent.

Not really possible, Linux doesn't know which window is in the foreground since it can't talk to the WM. But processes that just wait on I/O a lot will get a priority boost so they get to run more often, this usually works out to be the foreground app in X since they're always waiting on input from the keyboard and mouse.

During one of the slow phases do 'echo -n 'm' > /proc/sysrq-trigger' and look at/post the dmesg output.

 

[drag]

When I see problems with stuttering and poor performance the first thing I check out would be the DMA settings on your harddrives. If they arne't using direct memory access then overal system performance will be very slow.

Otherwise I pay close attention to dmesg output and top to see what apps iare consuming all the cpu.

 

[imported_LoKe]

More on DMA...

 

[Nothinman]

I wouldn't think it's DMA since he says "My Linux itself is actually quite fast, it's just the guest OS that gets the shaft."

 

[imported_LoKe]

Originally posted by: Nothinman
I wouldn't think it's DMA since he says "My Linux itself is actually quite fast, it's just the guest OS that gets the shaft."

You're probably right, but I figure it's a good thing to have on anyways, and since drag commented on it, I just followed up.

 

[xtknight]

I defragmented my virtual disk, which took a good amount of time, then rebooted, then started VMware, started XP. I used Linux a bit. Now (though with no apps open), my XP shuts down within 10 seconds. No surprise, since it's likely still in physical memory (or is it because I defragmented?) An HDtach under my guest OS in a good scenario reports a burst speed of 500M/sec (HDtach screenshot). That was during a fast time. But yes, when it gets swapped, that's where at least some of the problems come in.

Should I leave the page file under Windows enabled? I would if I were only running Windows, but isn't it better for Windows to address VMware and have VMware use Linux swap as it sees fit? That way I could get full control of just exactly how much memory is given out using VMware's settings, and it's easier to keep track of. Perhaps I'm just allocating too much memory to Windows. Right now I'm giving it 1024 MB. I'm used to going whole hog with 2 GB of physical RAM and one OS, but apparently those days are over.

How does DMA apply to SATA drives? Is it the same deal? I get this when trying to enable DMA on /dev/sda or /dev/sdb:

andy@ubuntu:~/vmware/Windows XP Professional$ sudo hdparm -d1 /dev/sda

/dev/sda:
setting using_dma to 1 (on)
HDIO_SET_DMA failed: Inappropriate ioctl for device

Wow, judging from the 'attach code' below, it looks like these files are in tons of fragments. My virtual disk is 20 GB using the split into 2 GB files option. Hopefully splitting reduces fragmentation but how do I defragment these? I already used the defragment function in VMware but they are still very fragmented. I've heard Linux file systems are almost immune to fragmentation or at least the performance degradation from it, but what about reiserfs?

My virtual disk and their parts are stored here on this reiserfs v3 (notail) partition.
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 180G 101G 79G 57% /home/andy
/dev/sda1 on /home/andy type reiserfs (rw,notail)

Swap partitions enabled (and in fstab) are:
/dev/sda6 29793 30300 4080478+ 82 Linux swap / Solaris (3.89 GiB)
/dev/sdb3 30002 30401 3213000 82 Linux swap / Solaris (3.06 GiB)

Should I disable sda6 since that is on the same hard disk as my VMware? I don't think I'll run out. Worst case scenario, 2 GB of my sdb3 swap will be used.

I think my hard disks are above average for speed, certainly not particularly slow from benchmarks of them I've read.

NF4 SATA Channel 1 (/dev/sda) - Maxtor [DiamondMax 10] 250G, 16M cache, 7200RPM (NCQ not enabled?)
NF4 SATA Channel 2 (/dev/sdb) - Western Digital [Caviar SE] 250G, 16M cache, 7200RPM (NCQ not available)

Does the nv_sata module enable NCQ by default? I could try that, despite hearing horror stories about my chipset and the DM10 HD with NCQ. I think mine was OK in Windows.

Apparently I don't have permissions to do:
sudo echo -n 'm' > /proc/sysrq-trigger
bash: /proc/sysrq-trigger: Permission denied

andy@ubuntu:/proc$ ls -al sysrq*
--w------- 1 root root 0 2006-06-28 22:44 sysrq-trigger

andy@ubuntu:/proc$ sudo cat /proc/sys/kernel/sysrq
1

Thanks for the guidance.

 

[xtknight]

Never mind, I had to use sudo -H -s to do the sysrq thing. Attached is results when both Linux and VM is fast.

 

[Nothinman]

How does DMA apply to SATA drives? Is it the same deal? I get this when trying to enable DMA on /dev/sda or /dev/sdb:

I don't think it's possible to use SATA without DMA and hdparm won't let you f' with it because the SATA drivers use the SCSI layer.

Does the nv_sata module enable NCQ by default? I could try that, despite hearing horror stories about my chipset and the DM10 HD with NCQ. I think mine was OK in Windows.

I don't think any drivers enable NCQ by default yet, maybe in 2.6.17 but more likely 2.6.18, but I'm not sure.

Never mind, I had to use sudo -H -s to do the sysrq thing. Attached is results when both Linux and VM is fast.

Well do it again when it's slow and we'll be able to compare.

Wow, judging from the 'attach code' below, it looks like these files are in tons of fragments. My virtual disk is 20 GB using the split into 2 GB files option. Hopefully splitting reduces fragmentation but how do I defragment these? I already used the defragment function in VMware but they are still very fragmented. I've heard Linux file systems are almost immune to fragmentation or at least the performance degradation from it, but what about reiserfs?

That filefrag output seems pretty bad, but I don't know if that's normal for reiserfs or not. Looking at a 8G set of files on XFS here I don't have any with more than 80 extents and only 4 of the 12 chunks are over 10 extents. I don't know of any defrag tools for reiserfs (XFS has xfs_fsr) so the only way you can probably defrag them is to backup/restore or at least copy them somewhere else and then back.

 

[Nothinman]

I just created a reiser3 filesystem on a file, put that VM in there and half of the vmdks are fragmented as hell, at least compared to the ones on the XFS filesystem. And that's after a fresh copy, so I don't think there's much you can do about that. And the VMWare defrag thing didn't do anything about it, not that I thought it would since I think it just copies the files around. But performance wasn't bad even running from the loopback filesystem, so I doubt that's your issue unless you just have a reiser3 memory leak or something.

 

[xtknight]

It is starting to get somewhat slow but not quite like yesterday so I may have to wait a couple days before everything gets fragmented/swapped enough. Only 3GB is being used today, vs. 4GB yesterday.

I disabled the Windows page file and the sda6 swap and also defragmented the virtual disk since with VMware. Linux is starting to get a bit slow now as well. Some keys are getting missed and there's lag when I backspace. Though this has always been happening after using VMware in the foreground for a while, or trying to boot it when it hasn't been "defragmented" in a while. Hopefully it's slow enough to conclude something (unfortunately for our debugging efforts it's getting faster as I speak). I will keep this arrangement open for a couple more days then do the sysrq again.

I am fairly sure that whatever the VMware "defrag" does, which takes a while, it fixes the horrible slowness problem. The rest is just the typical feeling of it reading off swap vs. physical which actually isn't bad. It seems to get swapped into physical fairly fast now and recover itself. When it was fragmented and I was trying to boot the guest OS, Linux would freeze for 20 seconds without letting me type. So there are times when both are very slow.

I have an empty 92G partition (on my slightly slower WD drive). Should I format it with xfs, put my VMware image there, and proceed, or should I let it run slow for a couple days first?

 

[xtknight]

OK, apparently I have accessed the hard disk enough for the guest OS to be unbearably slow by now. Here are the sysrq results. It's taking 3 minutes to get out of the screen saver. Load average hit 7, a little worse than yesterday.

 

[xtknight]

The 'heart attack' slow results are in. I hope they're useful because it took me five minutes just to get this post in without my keyboard stuttering. :laugh:

 

[Nothinman]

I am fairly sure that whatever the VMware "defrag" does, which takes a while, it fixes the horrible slowness problem.

Then I would definitely try another filesystem. I've never been a fan of reiserfs and it's known for being fast with lots of small files, not a few large ones.

I have an empty 92G partition (on my slightly slower WD drive). Should I format it with xfs, put my VMware image there, and proceed, or should I let it run slow for a couple days first?

Couldn't hurt to try.

The sysrq memory output doesn't look all that different, just comparing the numbers do you see any that are drastically different between them?

 

[xtknight]

Originally posted by: Nothinman

I am fairly sure that whatever the VMware "defrag" does, which takes a while, it fixes the horrible slowness problem.

Then I would definitely try another filesystem. I've never been a fan of reiserfs and it's known for being fast with lots of small files, not a few large ones.

I have an empty 92G partition (on my slightly slower WD drive). Should I format it with xfs, put my VMware image there, and proceed, or should I let it run slow for a couple days first?

Couldn't hurt to try.

The sysrq memory output doesn't look all that different, just comparing the numbers do you see any that are drastically different between them?

I don't know. It's greek to me but it looks like 'active' is now a lot higher than when it was fast. Probably because I loaded another VM. (A while ago I formatted the partition as XFS, I am now creating an XP64 virtual machine on the xfs, it's 20 GB in size and pre-allocated. After the preallocation and the start of installing XP64, all hell broke loose.)

P.S. FireFox crashed during typing my last post (though it has randomly before) and VMware came to focus in my face for no apparent reason, so maybe all is not perfectly stable. The really slow results are in my last post. This is after running 2 VMs with 1024MB allocated each. Basically it takes 5 minutes to do anything in Linux, the mouse stands still for 5 seconds at a time and jitters around, and nothing is remotely usable.

 

[xtknight]

So, just to make everything clearer. Right now, this is what I had open when everything was terribly slow:

Firefox with lots of tabs open (which crashed as I was writing a post)
Lots of terminals
IM Client
XMMS
VMware
[*]Running: Windows XP Professional (32-bit): 20 GB virtual disk on reiserfs. Not pre-allocated, but recently defragmented, allocated 1024 MB of RAM. Inside XP32, lots of various apps open like PhotoShop, DreamWeaver, IE, Opera, Firefox.

[*]Running: Windows XP Professional (64-bit): 20 GB virtual disk on xfs. Pre-allocated 20 GB, allocated 1024 MB of RAM. I was starting the XP64 install.

After it started the Windows blue installation screen, Linux got extremely slow and that's when I did my last sysrq dump. The mouse was stuttering and it was taking 5 minutes for a sentence I typed to echo out. Now I am in the same state 10 minutes later, except for a new FireFox instance. Everything seems perfectly fine now as Windows XP 64 is still installing in the background.

 

[xtknight]

I noticed something.

Slow: Active:343000 inactive:127420 dirty:13434 writeback:0 unstable:0 free:4097 slab:10208 mapped:394739 pagetables:5784

[77532.849049] Swap cache: add 563913, delete 546903, find 52203/66203, race 0+0
[77532.849051] Free swap = 1412900kB
[77532.849053] Total swap = 3212992kB
[77532.849055] Free swap: 1412900kB

Fast (I did sysrq again 20 mins later): Active:341744 inactive:126279 dirty:10 writeback:0 unstable:0 free:4212 slab:10731 mapped:338789 pagetables:5100

[79326.308879] Swap cache: add 1316743, delete 1263760, find 331702/433744, race 0+0
[79326.308882] Free swap = 2455524kB
[79326.308883] Total swap = 3212992kB
[79326.308885] Free swap: 2455524kB

It looks like a gig of swap was released somehow (and "dirty" is a lot lower)?

 

[Nothinman]

P.S. FireFox crashed during typing my last post (though it has randomly before) and VMware came to focus in my face for no apparent reason, so maybe all is not perfectly stable.

It's possible that was the OOM killer trying to free memory, did you see anything in the kernel logs from it?

This is after running 2 VMs with 1024MB allocated each.

Didn't you say you only have 2G of memory? Why would you expect your system to run well with 2 active VMs allocated all of your memory?

 

[xtknight]

When my memory was maxed out on Windows, I'm pretty sure I could still do stuff without it freezing my mouse and keyboard for 3 minutes at a time, but I suppose different OSes handle things different ways. BTW, this happened just as much with only one VMware open at 1024M for three days and nothing excessively huge open under GNOME except for a couple 20-tab Firefoxes/IM Client/terminals. Even yesterday, 4G of virtual memory was allocated (GNOME+apps and one 1024M VM running). My XP32 had horrible fragmentation problems judging from that output.

The reason I no longer think it's simply 'too much memory'? Because I've booted the virtual machine on a clean Linux boot, and it still stalled my Linux awfully until I used VMware's "defragment".

Windows XP Professional x64 Edition Fast-flat.vmdk: 9 extents found
And this is a 20G file! (Though it has only been installed for a few hours. Hopefully it won't get much worse, or if it does, I can defragment with xfs.)

I've rebooted since the last horrible slow-down (so I can't get that kernel log), and I've been real careful since. FireFox (well SwiftFox) has spontaneously quit for no reason before.

What I've decided to do is use my XP64 (over xfs) exclusively and no longer use the XP32. I've finished migrating my files/software so I will simply no longer use the XP32 that got very bloated and split up. The XP64 is already working a lot better and faster, and this time I left the page file on and allocated it 640M via VMware. Thanks for the help/commands. XFS also seems to be fast, so I'll certainly leave my 92G partition as xfs. I just use ReiserFS because that's what I've always used and I thought it was the fastest FS out there.

My conclusion is that XP32 got very fragmented (plus the NTFS system inside it), and it was on the same physical hard disk as my Linux. And I was just giving it too much memory. My XP64 is now on a separate hard disk and only given 640M of memory (plus the 1.5-3.0G it can get itself by using its page file).

 

[Nothinman]

When my memory was maxed out on Windows, I'm pretty sure I could still do stuff without it freezing my mouse and keyboard for 3 minutes at a time, but I suppose different OSes handle things different ways.

Yea, technically Linux shouldn't do that either but Windows puts a lot more emphasis on the UI since they're more tightly integrated. IMO Linux handles it better overall though, on my XP machine if I even start 1 VM with 256M allocated to it things start paging like hell, it's very annoying.

I just use ReiserFS because that's what I've always used and I thought it was the fastest FS out there.

Supposedly it's fast with lots of really small files but at the expense of CPU time IIRC. And I don't trust reiserfs, every time I've tried to use it it's ended in dataloss or at the best a backup/restore to another filesystem. That and Hans Reiser is an ass.

My conclusion is that XP32 got very fragmented (plus the NTFS system inside it), and it was on the same physical hard disk as my Linux. And I was just giving it too much memory.

The two former points shouldn't matter, it was most likely just the latter. Unless reiserfs is just really bad handling the VMWare's usage pattern.

 

[xtknight]

It's been up for a day, and the XP64 20GB (xfs) virtual disk is still in 9 extents. That's the only file on the partition at the moment. Screen saver exits in less than 1 sec after being on all night. That's a record. Amazing what a different file system can do. Windows performs awesome too with 640M allocated and it having its own page file within the 20G vdisk. I'm fairly certain the Windows kernel writes half of things to physical memory and half to page file at once, so for doing that it's doing great.

I guess I'll leave my / partition and /home partitions as reiserfs for now (it's too much work to convert them), but in the future I'll likely use ext3 or xfs instead. Actually converting my /home partition wouldn't be too hard since I have ~72G free and only ~26G used on my /home. I'm going to do that in fact.

I read somewhere that 'cp' is not reliable for disk mirroring (someone on the ubuntuforums cringed at the idea) and I should use 'rsync -aS' instead. Why is that? Won't "cp -rf" copy everything? That's what I used to move my home directory before and nothing's gone missing as far as I can tell. Any difference between 'cp -rf' and 'cp -a'?

I often find Windows crying that something is using a file when I try to copy it. Will Linux do the same or should a 'cp -rf' of the home dir while GNOME is running be fine?

 

[Nothinman]

Windows performs awesome too with 640M allocated and it having its own page file within the 20G vdisk.

I usually only give Windows 265M and it runs fine in VMWare.

I read somewhere that 'cp' is not reliable for disk mirroring (someone on the ubuntuforums cringed at the idea) and I should use 'rsync -aS' instead. Why is that? Won't "cp -rf" copy everything? That's what I used to move my home directory before and nothing's gone missing as far as I can tell. Any difference between 'cp -rf' and 'cp -a'?

It should be fine, the only thing I would worry about is symlinks but I think it'll handle those fine too. If you're really worried you can use tar or cpio to copy the files.

http://www.greenfly.org/tips/filesystem_migration.html

I often find Windows crying that something is using a file when I try to copy it. Will Linux do the same or should a 'cp -rf' of the home dir while GNOME is running be fine?

It'll be fine, unix file locking is much more lenient than that of Windows.