Why is my page file being used so much?

Page 2 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

Phoenix86

Lifer
May 21, 2003
14,644
10
81
Originally posted by: STaSh
OK, I'll bite... what? Perhaps I misused a term? I'm aware of other VM files for things like system cache, but those are seperate files from the PF.

Virtual memory is not a file, it's an abstraction. Hence the word 'virtual'. It is not RAM+PF. VM is the address space available to any process on a system. On a 32-bit system, the VM will always be 4 GB.

In Windows, this space is split between user and kernel mode processes, each having 2 GB available (by default).

I'm aware it's more than a file (RAM isn't a file), it's a pool of memory between files and RAM. I'm aware of the addressable limits2/4GB. What's a better term for the section of VM that is PF+RAM (represented by commit charge)? Perhaps VM is just too broad of a term there... I do note in the writeup that VM isn't just the PF and RAM, I'm not trying to say that disabling the PF stops VM, by any means. Just that it limits where a large amount of data gets paged.
 

Phoenix86

Lifer
May 21, 2003
14,644
10
81
But the fact remains that unless you never install any updates or new software you can't be sure that the sizes you picked will remain constant and the cost of leaving the pagefile growable is virtually nothing.
Good point. I do note to leave breathing room (25% higher than your current requirements) in the calculations, however it should be retested over time. Perhaps I should note to re-evaluate your requirements after major system changes, like SP2 for example, or every few months.

However, if you do retest, you will find out your new memory requirements... The argument can be made, that if you ever run out of memory, you are not within the forumula I layed out.

In many situations even if you set the pagefile to 0 the system will create a small (~20M) pagefile on the system drive because XP knows that it's necessary in some cases and having one is safer.
I have not seen this on any system (XP) I have disable the PF on, is it a seperate file from pagefile.sys? Got a link about this?

Data is always paged to physical memory whether you have a pagefile or not, removing the pagefile should not decrease the amount of paging on a system at all. Well unless you cause things to die because there's not enough memory to handle something and there's no pagefile to free some of memory.
OK, there is a disclaimer about this in my post, as I said, there is a lot of discussion not having a PF. Sure, data is always loaded into RAM first, but then what happens when new data is loaded, and there is still enough RAM for both? Often the system will page some of the original data to the PF, which isn't necessary (there is still room in RAM). However, the system will try to keep RAM free so you can load new data into it. That's the problem, **if** you have enough RAM to run everything, you never need to page to disk.

A memory dump from a BSOD is stored in the pagefile and then copied to a seperate dump file, if you want a full dump you need a pagefile the size of physical memory + some extra (20M IIRC) but if you want a kernel dump you only need like 12M. The only thing a full dump will give you is all of the data on running processes, it's possible one of them is causing the BSOD but not likely and you can usually identify the driver at fault from just a minidump.
Sounds like a minidump would be enough. Most BSODs are bad hardware or bad drivers. Anyways, I have never NEEDED a memory dump analyed to solve a system's problems. Probably would help in TSing, but it's not 100% necessary...

And the hiberfil.sys file is irrelevant, it stores system state so that it can be resumed from later and it has to be the size of physical memory because there's a chance that you would try to hibernate the system when memory is 100% full and part of the pagefile is used and unless your pagefile is 2 or 3 times as big as your physical memory the chances are slim that you'll have enough space in the pagefile to hold the memory state and the current data in the pagefile.
My point about hiberfil.sys is, there could be another file like that, not the PF, where data is dumped. That way it's not dependant on having a PF.
 

Nothinman

Elite Member
Sep 14, 2001
30,672
0
0
OK, there is a disclaimer about this in my post, as I said, there is a lot of discussion not having a PF. Sure, data is always loaded into RAM first, but then what happens when new data is loaded, and there is still enough RAM for both? Often the system will page some of the original data to the PF, which isn't necessary (there is still room in RAM). However, the system will try to keep RAM free so you can load new data into it. That's the problem, **if** you have enough RAM to run everything, you never need to page to disk.

If there's enough room for both there will be no data put in the pagefile unless the system decides that something in memory hasn't been touched in so long that it would be more beneficial to remove it and replace it with something else like system cache. And I would guess that most of the time that data is executables or shared libraries which don't get put in the pagefile anyway.

I have not seen this on any system (XP) I have disable the PF on, is it a seperate file from pagefile.sys? Got a link about this?

http://support.microsoft.com/d...5270&Product=winxp

My point about hiberfil.sys is, there could be another file like that, not the PF, where data is dumped. That way it's not dependant on having a PF.

Theoretically, sure. But there isn't one and if there was it would just be a pagefile by another name.
 

Phoenix86

Lifer
May 21, 2003
14,644
10
81
If there's enough room for both there will be no data put in the pagefile unless the system decides that something in memory hasn't been touched in so long that it would be more beneficial to remove it and replace it with something else like system cache. And I would guess that most of the time that data is executables or shared libraries which don't get put in the pagefile anyway.
Can you point me somewhere to do some reading about what gets paged/when/why? This is definatly one of my weak spots in understanding VMM.

From experience, I have notices systems paging to disk when it doesn't seem they need to. IE, I have say 512MB RAM, and I'm using ~350MB, yet the system is busy with heavy activity on the PF. Disabling the PF makes this "problem" go away, if you have enough RAM.

http://support.microsoft.com/d...5270&Product=winxp

I have run into this problem before (windows 2k and NT). It's a situation where there isn't enough memory to load the OS. Set a machine up with np PF and 64MB RAM, and you will get this error. Change the 64MB to 256MB and it goes away.

This is no different an error than what you get when you try to load a program and don't have enough memory. However, on boot it will create a temporary PF so you CAN boot to fix the problem. This is a step up from NT, which simply wouldn't boot.

The article doesn't state, given enough memory, that this will happen. This will only happen when you don't have enough memory, which means you are outside my sizing forumula.

Theoretically, sure. But there isn't one and if there was it would just be a pagefile by another name.
Yes, this is a "if I had it my way" kinda thing. The difference is, one doesn't require a PF, meaning those running sans PF could still get the same error reporting level. Remember, "no PF" is a MS setting, they should assume some people are going to run their systems this was, and should reduce dependencies on this file where they can.
 

Nothinman

Elite Member
Sep 14, 2001
30,672
0
0
Can you point me somewhere to do some reading about what gets paged/when/why? This is definatly one of my weak spots in understanding VMM.

Executables and shared libraries don't get put in the pagefile because they already have a backing store on disk in their respective files. If the memory being occupied by an executable is needed for something else it's just cleared and reused, if the executable is needed again it's just paged back in from disk like it was when you ran it the first time. I don't know all of the Windows corner cases though, most of my VM knowledge comes from Linux and MS's documentation doesn't seem as complete when it comes to the internals of their OS and I don't feel like paying for a copy of "Inside Windows 2000".

Yes, this is a "if I had it my way" kinda thing. The difference is, one doesn't require a PF, meaning those running sans PF could still get the same error reporting level.

I guess you mean a file just for memory dumps? Sure it might happen but I'm sure it's not high on their list of priorities and at the earliest it won't happen until Longhorn.
 

CTho9305

Elite Member
Jul 26, 2000
9,214
1
81
So to summarize, you're arguing that if you just track your memory usage for a while, you can quickly estimate a good page file size (RAM + PF should be usage + 25%), right? Why do you think Windows isn't smart enough to manage that on its own? I've noticed after repeatedly using a lot of memory that at bootup, my commit charge limit starts out higher. Windows did that for me - I didn't have to mess with any settings because my usage patterns changed to be more memory-intensive.
 

Phoenix86

Lifer
May 21, 2003
14,644
10
81
I'm not sure it's entirely fair to apply Linux VMM to Windows. I'm sure they are comparable, but I have seen differences mentioned.

I never really though about exes and libs being put into the PF, it makes perfect sense to use the file on disk instead of creating another copy in the PF... Guess that book has the best answers, as it's been mentioned in other threads.

I guess you mean a file just for memory dumps? Sure it might happen but I'm sure it's not high on their list of priorities and at the earliest it won't happen until Longhorn.
Yes, that's exactly what I mean. Hmm, I'll have to suggest that.

CTho9305, several reasons.

First, page file fragments. When you PF expands, do you where on the HDD do you think it appends the data to? A. Anywhere it has space. It will not rebuild the file into a single, unfragmented file. Now, of course all you have to do defragment that file right? No. You can't. At least not without 3rd party tools. Now, do this a few times and you will "feel" it when paging to and from disk because of access time. Do this on a 1GB file, and you will really "feel" it. Of couse, there are ways around that (seperate partition), but most single HDD systems aren't setup that way.

Second, access times. Why access a 1GB file, or if you believe the 1.5-2.5 times RAM equation and have 1GB of RAM, 2.5GB file, when all you need is 300MB, or 0?

Third, who says VMM is 100% effecient? Not me. As I stated above, I have seen systems page to and from disk when there is absolutly no reason to. IE all programs running will fit in RAM alone, so why is the system even accessing the disk? Because, and here's a bit of speculation based on experience, it's trying to keep as much RAM free as possible. If you do need it, it's available. What it doesn't know is the limits of the system. This is why it's not effecient, it's flexible.

If it weren't flexible(expands PF as needed, keeps RAM free), it would crash systems with out of memory errors, not sure why it tries to keep RAM free... Probably assuming your going to load something else, and by having things paged to disk earlier than necessary (IE it pages less used items) it's ready. That assumes a few things, one that you are going to load something else (only you can answer that) and that you may exceed your current limit (again, only you can answer this).

The reason windows CAN'T do it's job effectivly is because it doesn't know YOUR limits. How much memory is required when the user launches X,Y, and Z applications? Will you also launch A,B, and C applications too?!? It doesn't know, but you do, and you can measue it.
 

Nothinman

Elite Member
Sep 14, 2001
30,672
0
0
I'm not sure it's entirely fair to apply Linux VMM to Windows. I'm sure they are comparable, but I have seen differences mentioned.

I'm sure there are differences, but the beast is in the implementation of the basic concepts. Things like how does XP scan the page tables looking for pages to page out and how does it map shared pages back to a process if it does at all.

Guess that book has the best answers, as it's been mentioned in other threads.

I'm sure it does, I've just never had a real reason to buy the thing since I tend to avoid Windows in general and there's virtually no chance I'll be doing kernel or even driver development for Windows. Since I do use Linux a lot and I can poke around in the kernel really easily.

Because, and here's a bit of speculation based on experience, it's trying to keep as much RAM free as possible. If you do need it, it's available. What it doesn't know is the limits of the system. This is why it's not effecient, it's flexible.

It's not trying to keep memory free, it's trying to speed things up by using that memory for system cache instead of something that hasn't been touched in an hour. The idea is that you havn't used that data in so long that it'll probably be more beneficial to use that memory for something that you're actually using.
 

Phoenix86

Lifer
May 21, 2003
14,644
10
81
Originally posted by: NothinmanIt's not trying to keep memory free, it's trying to speed things up by using that memory for system cache instead of something that hasn't been touched in an hour. The idea is that you havn't used that data in so long that it'll probably be more beneficial to use that memory for something that you're actually using.
OK, same result different means. If it doesn't need to page data to disk because there is enough in RAM, it still does. I see it's trying to help, and on lower memory systems, this is good. However, I just don't think VMM handles high amounts of RAM as well as it could if limited. This logic in VMM is the reason I wrote the article.
 

Nothinman

Elite Member
Sep 14, 2001
30,672
0
0
It's the same logic that Linux uses and noone complains about swap usage on Linux (minus a few badly tuned kernel releases that were fixed quickly). For example, my laptop has been booted for roughly 7 minutes and half my memory is being used for system cache.

~# free -m
total used free shared buffers cached
Mem: 1010 854 156 0 12 518
-/+ buffers/cache: 323 687
Swap: 1027 0 1027
 

Phoenix86

Lifer
May 21, 2003
14,644
10
81
No one complains about it, so there isn't a problem? Err... no.

The PF isn't cache, it's a poor substitute. Cache supposed to be faster than the component it's caching. IE, HDDs cache is faster than the HDD. L1-2-3 cache is faster than RAM. Yet, when applied to the OS, the HDD caches RAM. :confused:

Let me ask you this, since I guess these Qs are kinda OS neutral... *admits VERY little linux knowledge, so feel free to slap me around on any linux specifics*

On your linux box, how much RAM do you have, and how much memory is your system using? If RAM is not larger than memory requirements, assume it is, now, why are you caching RAM to the HDD? Why empty RAM of data if you have enough for everything you need to run? Could my page file sizing forumla apply to linux just as well as windows (given you take the memory measurements in whatever comparable linux tool)?
 

Nothinman

Elite Member
Sep 14, 2001
30,672
0
0
No one complains about it, so there isn't a problem? Err... no.

Well with the amount of windows users that complain, you would think someone on the linux side would complain too if there was an issue. And that works both ways, just because people complain about something that doesn't automatically make it wrong.

The PF isn't cache, it's a poor substitute. Cache supposed to be faster than the component it's caching. IE, HDDs cache is faster than the HDD. L1-2-3 cache is faster than RAM. Yet, when applied to the OS, the HDD caches RAM

The PF isn't trying to be cache, it's temporary storage for things that aren't already found on another medium.

On your linux box, how much RAM do you have, and how much memory is your system using?

My notebook here has 1G total with 993M used and 598M of that is system cache. I know the formatting below will get screwed up, but I can't remember the code tags.

free -m
total used free shared buffers cached
Mem: 1010 993 17 0 15 598
-/+ buffers/cache: 379 631
Swap: 1027 0 1027

If RAM is not larger than memory requirements, assume it is, now, why are you caching RAM to the HDD?

I'm not, I'm caching filesystem data to memory.

Could my page file sizing forumla apply to linux just as well as windows (given you take the memory measurements in whatever comparable linux tool)?

The same ideas apply to Linux, yes. But the main difference is that Linux wasn't designed to rely on swap space, in fact if you want you can compile the kernel without support for swap partitions completely, mainly for embedded use.
 

Phoenix86

Lifer
May 21, 2003
14,644
10
81
I'm not, I'm caching filesystem data to memory.
OK, why store temporary data on a slower medium? If the data is going to get called, why not have it in RAM vs HDD, again, assuming there is enough RAM?

I just don't see a justification for a PF on systems with "enough" RAM. I see more and more machines being built with a LOT more RAM compared to requirements vs. 5 years ago when a PF was required on all systems. IE 5 year old machine may have 128 MB RAM, but uses x2 that. Now it's common for systems have 1GB of RAM, and only use 1/2-2/3 of it.
 

Nothinman

Elite Member
Sep 14, 2001
30,672
0
0
OK, why store temporary data on a slower medium? If the data is going to get called, why not have it in RAM vs HDD, again, assuming there is enough RAM?

That's what's happening, the cache line in the 'free' output is how much data from the hard disk is being cached in memory, which in the above example is 598M.
 

Phoenix86

Lifer
May 21, 2003
14,644
10
81
All this being said, I don't see how the same wouldn't be true in a linux environment... It would seem those OSes could benefit from this as well.

BTW, I often hear about this "complaint" from gamers, who move/use lots of data which results in more disk paging (not trying to say linux users don't move/use lots of data, by any means). I wouldn't be surprised if this isn't noticed as much because of the applications running. IE, who cares if the system slows down for a few seconds when it's paging to disk unless your playing a game where you get pissed about: load times and FPS.
 

Nothinman

Elite Member
Sep 14, 2001
30,672
0
0
All this being said, I don't see how the same wouldn't be true in a linux environment... It would seem those OSes could benefit from this as well.

Essentially if you do benefit from not having a pagefile (or swap space in unix terms) there's another problem, possibly with the tuning of the VM system you're running. The pagefile is used to store data when there is things that the OS has determined need to be in memory. That determination could have come from "Those pages havn't been accessed in X million ticks, it would be better to use that memory for system cache instead of wasting it on unused data" or "I need to page in that executable you just clicked on but there's no memory available so I'll need to put this program's data into the pagefile to make room". The latter is a more extreme case though because generally the system cache will be shrunk to make space before anything is actually put into the pagefile.

There was a big discussion on lkml a while back about how swap space helps performance, but it's hard to find since I can't remember anything unique that came up in there. There's a more recent one with the subject of "VM Vs Swap Space" that's not too bad, but it's extremely short.

IE, who cares if the system slows down for a few seconds when it's paging to disk unless your playing a game where you get pissed about: load times and FPS.

I would notice and I do care because I run VMWare every day and VMWare is a memory hog. Right now VMWare has 216M in memory just for itself, incidentally Thunderbird also has 208M in memory.
 

CTho9305

Elite Member
Jul 26, 2000
9,214
1
81
I would notice and I do care because I run VMWare every day and VMWare is a memory hog. Right now VMWare has 216M in memory just for itself, incidentally Thunderbird also has 208M in memory.
It's worth noting that if you have an application that leaks memory, the leaked data NEVER gets touched once it leaks. If you have no swap file, it has to stay in RAM. If you have a swap file, it can be paged to disk (since it will never be touched again) and anything it gets replaced with is better than leaked data.
 

exodus454

Senior member
Apr 11, 2004
465
0
0
Originally posted by: Nothinman
Certain games, especially since they're getting more and more memory hungry. I also believe Photoshop will fail without a pagefile. Generally anything that makes use of anonymous memory mappings, keeps lots of data in memory, etc has a chance to fail because there is no other backing store available. For that reason it's really difficult to disable the pagefile, usually if you try it'll creage a small one (~20M) in the windows directory without telling you.

photoshop runs fine with no pagefile, it will just give you an error on starting the program that the pagefile has been disabled and asks you if you want to continue or not. I have 512mb memory and never had a problem with running no pagefile.
 

kylef

Golden Member
Jan 25, 2000
1,430
0
0
First of all, I think we can all agree that the original poster was confused about the "PF Usage" bar graph in the Performance tab of Task Manager. He thought that meant Windows was literally using that much space in the pagefile. It is not. To my knowledge, there is no easy way to figure out exactly how much of the Pagefile is actually in use at any given time. Task Manager certainly doesn't show this, and neither does Perfmon. If anyone knows of an easy way, please enlighten me. I'll do some research on the subject and try to get back to everyone too.

Originally posted by: Phoenix86
I never really though about exes and libs being put into the PF, it makes perfect sense to use the file on disk instead of creating another copy in the PF... Guess that book has the best answers, as it's been mentioned in other threads.
It doesn't... When we say that mapped files are "paged to disk", we mean simply that they are written back to their original storage location (the disk file itself, NOT the pagefile). This is true for ANY file I/O access in Windows, including both EXEs and regular files. Your pagefile does not contain memory-mapped mapped files. Your System cache does. When a page in the cache needs to be flushed to disk, then the Cache Manager figures out how to write it back to disk: to the disk file if it's a memory-mapped file, or to the pagefile if it's just a random dirty page from some process.

I guess you mean a file just for memory dumps? Sure it might happen but I'm sure it's not high on their list of priorities and at the earliest it won't happen until Longhorn.
Yes, that's exactly what I mean. Hmm, I'll have to suggest that.
No, there is a reason dumps are written to the pagefile. It is mounted specially by the OS so that file system filter drivers (such as virus scanners) play no part in accessing it. That way, if kernel memory is corrupted, it increases the chances that a dumpfile will be written to disk correctly. (Imagine if the file system filter driver were the cause of the system crash in the first place: you'd have a problem trying to write the dump file through the file system, wouldn't you? That's the advantage of the pagefile.)

Third, who says VMM is 100% effecient? Not me. As I stated above, I have seen systems page to and from disk when there is absolutly no reason to.
Again: paging to and from disk can be two different things: 1) Pagefile activity and 2) File cache activity. If you use Perfmon to track Page faults/sec, the faults are triggered by ALL reads to memory-mapped files by any process on your system. Since all file I/O is implemented as memory-mapped files behind the scenes, that means that ALL file I/O reads that are cache misses show up as a system page faults! (Note that unless the file is explicitly memory-mapped by a process, the process is not charged with the page fault.)

In other words, there is a good chance that the "disk activity" you're seeing is not Pagefile activity at all, but simply Cache Manager activity.

That said, WindowsXP has lots of background services running (many under the "svchost.exe" banner) that can occasionally use up lots of RAM temporarily. The Cache Manager ALWAYS has a good reason to evict pages to the pagefile. And it always attempts to evict pages in LRU (least recently used) order to minimize the resulting performance penalty.

Because, and here's a bit of speculation based on experience, it's trying to keep as much RAM free as possible. If you do need it, it's available. What it doesn't know is the limits of the system.
Yes, the Cache Manager does perform some "balancing" to keep some pages on the free list for programs that request memory. But it doesn't do this randomly, and it absolutely knows what the limits of the system are. I will do some research to find out what the "balancing" criteria are.

The PF isn't cache, it's a poor substitute. Cache supposed to be faster than the component it's caching. IE, HDDs cache is faster than the HDD. L1-2-3 cache is faster than RAM. Yet, when applied to the OS, the HDD caches RAM.
Hmm... based on that statement, it doesn't sound like you understand the concept of a memory hierarchy. Every component in a memory hierarchy "caches" the lower-level hierarchy. Some of the levels are exclusive (or "orthogonal") while others replicate everything in the higher levels ("inclusive").

The top level is CPU Registers. The second level is L1 cache. The middle levels are the remaining L? caches (L2, L3, L4...). AMD, for instance, uses an orthogonal L2 cache, while Intel does not: it's implementation-specific. Next you get to system RAM. System RAM is always inclusive; i.e., you never see something higher in the cache hierarchy that isn't also stored in RAM. Finally, you get to disk, which is orthogonal: it is ONLY used when there isn't enough room in the higher levels to store all data.

So your claim that in the OS "the HDD caches RAM" is simply not correct. The higher levels are caching the lower levels, throughout the whole computer. The pagefile is only used when there isn't enough room in System RAM to store all of the data requested by processes. It's used as a "backup", not a cache. The System Cache Manager determines when it must flush some pages to the pagefile, and it always has good reasons to do so.

Does that clear things up at all? Or have I simply opened up more cans of worms?
 

n0cmonkey

Elite Member
Jun 10, 2001
42,936
1
0
Your ram is going to get used up to the best of the OS's ability whether you have a pagefile or not, whether you let the OS handle your pagefile or not. When considering this issue, maybe everyone should ask themself two things: Do you want the ram to be filled with useless application data or filecache which might speed up some operations? Do you want to deal with potential problems arising from thinking you understand the subsystems involved better than <insert OS vendor>?
 

Phoenix86

Lifer
May 21, 2003
14,644
10
81
Originally posted by: kylef
Does that clear things up at all? Or have I simply opened up more cans of worms?
Probably the latter. :) The more I learn, the more I realize there is to learn; everytime I learn something, I know less.
 

kylef

Golden Member
Jan 25, 2000
1,430
0
0
Originally posted by: Phoenix86
Originally posted by: kylef
Does that clear things up at all? Or have I simply opened up more cans of worms?
Probably the latter. :) The more I learn, the more I realize there is to learn; everytime I learn something, I know less.
There is always more to learn, but that doesn't stop me from trying to learn it all. :)

I apologize if my description was convoluted earlier... when I describe things, I have a tendency to assume the reader already understands some pretty advanced OS concepts. (Some concepts aren't really that advanced, they're just foreign to non-Windows tech heads.) I am patient, however, so if you have questions I will usually respond when I have time.

I still haven't found out how to monitor pagefile usage in Windows. As soon as I find something out, I'll respond back to this thread, because it seems germane to the topic.