C# how to prevent outofmemoryexeption

[Red Squirrel]

C# seems to really not like storing large amount of data in a string, regardless of how much memory is available.

At random, if I try to store say, 100K of string data in a string, or even less, it may crash.

Other then "don't store that much data in a string" what is a way around this limitation? I know it has to do with the stack getting fragmented but I mean, C++ can hold GIGS of data in a string, so there's got to be a way around this limitation.

If you're wondering, this is for a program that used to write this data to file in buffers, but I am converting it to SQL. Some items are very large so they need a very large string in order to be stored/restrived from SQL. These strings are more or less used as buffers, then emptied out when done. I would not say it's the BEST way, but if I can get around this limitation, it will work well.

 

[tfinch2]

Are you using String or StringBuilder?

EDIT: Also, http://www.pcreview.co.uk/forums/thread-2539908.php

 

[Red Squirrel]

Just string. Would StringBuilder be more efficient? I was wondering that. I HAVE to use string when pulling out of the DB since that's what the function returns, but I could always convert it right away and use StringBuilder everywhere else.

And yeah I saw that thread too. In my case there's not too much concatenating being done. It's basically stored as is and left that way.

 

[degibson]

Is your string on the stack or on the heap?

 

[Red Squirrel]

How would I tell that? Any way to force it on the heap? That would probably be more efficient for my needs as the stack is more limited.

 

[nickbits]

is this a standalone or web app? if web app check your app pool max size.

i don't think 100k should be a problem in general.

are 100% sure that is causing the problem and is not just the symptom of a different problem?

 

[Markbnj]

Originally posted by: RedSquirrel
How would I tell that? Any way to force it on the heap? That would probably be more efficient for my needs as the stack is more limited.

All reference types are managed heap objects in C#.

http://msdn.microsoft.com/en-us/library/490f96s2.aspx

 

[Red Squirrel]

I'm pretty sure it's that. The error happens during load, and this is the most memory intensive part where these strings are used. Basically it's a game server. Each item in the game is saved as 1 SQL row which is a stream of characters. Houses are the biggest as they are composed of many wall components. When it crashes, it's always while loading the same house (there's this one house that is about 84k chars saved). During that process it's handling the string then populating other variables, so adding to the memory just seems to put it over the top. I have about a gig free at that point, and that's not counting the swap file.

This is a C# app running on Windows 2003. Anything in win2k3 I can do to optimize memory use?

 

[nickbits]

You could add some GC.Collects() into your loading loop.

The issue is probably that it can't find 100k of continuous memory not that there isn't 100k available in total.

 

[imported_Dhaval00]

Originally posted by: nickbits
You could add some GC.Collects() into your loading loop.

The issue is probably that it can't find 100k of continuous memory not that there isn't 100k available in total.

Don't do this... it will only worsen the situation.

String is indeed a reference type (heap), but it is a very special reference type within the CLR realm. There are many optimizations built around the String type... so much so that its manipulation is comparable to value types.

Whenever dealing with substantial amounts of data and complex string operations, use the StringBuilder type. Even within the StringBuilder type, there are multiple methods to be exploited like AppendFormat.

As a side note, what kind of operations are you doing on this string?

 

[Red Squirrel]

Not too many operations are done to it. Here's how a world load works:

A "SELECT col,col2,col3 FROM items;" query is made
col, col2 are rather small and non issue here, they're used and discarted

col3 can be anywhere from 100 to 100k bytes, but it's somewhat controled. Ex: I should not ever see it go to like 1 MB. In my current DB the biggest one is 80K

This data is stored into a new string.

A new object of various type is created, and then a special "Loader" object is created where this object is put inside, and the string is put inside, using 2 arguments. So at this point there is 2 copies of the same string, but then on the next fetch row it gets overwritten.

Once all the loader objects are created (each one containing a string of data) then they are looped through and a "serialize" function is called. This function goes through the string byte by byte and interprets the data, then populates the variables in the object that was stored in the loader object.

It's During this serialization portion where it will normally crash.

Once the world is loaded, these long strings are no longer needed.

However, they do get recreated to save changed data, then submitted as a UPDATE or INSERT query, throughout the program's execution, but so far this process has not caused any problems. The DB updates are done in very small batches at a time.

 

[degibson]

"C#" -- sounds more like "Cb" to me.
Real programmers use .asciiz

 

[imported_Dhaval00]

Can you post sample code? It is possible that by the time you reach the serialization logic, you've used too much memory...

Also, have you tired this:

Originally posted by: nickbits
is this a standalone or web app? if web app check your app pool max size.

Also, to clarify, is this Website running within its own AppPool?

 

[Red Squirrel]

It's a C# app, not a web app. I highly doubt I'm using all memory as I was watching in the process manager and it was at around 400K around that time. Iti's also hit or miss. On my XP workstation where I do my coding, it never did it, but on the test server, it did it. The test server has 2GB of ram and is dedicated to this app while my XP workstation is 3.5 but has more stuff open.

I'm not home now but I can post sample code later. Maybe there's a way I can make this more efficient.

I was also thinking of using some kind of internal compression but that probably would not help as it still needs to decompress at some point.

 

[Ken g6]

I just happened to recall that Java has a memory limit on its virtual machine that is much smaller than the real machine's memory. It looks like the same may be true for ASP.NET as well. It might be true elsewhere, and I'm just not searching with the correct Microsoft terminology.

 

[Crusty]

Don't forget strings are immutable in C#, so if you are performing lots of operations on the strings you'll be wasting lots of memory and time copying the string values around.

 

[imported_Dhaval00]

Originally posted by: Ken g6
I just happened to recall that Java has a memory limit on its virtual machine that is much smaller than the real machine's memory. It looks like the same may be true for ASP.NET as well. It might be true elsewhere, and I'm just not searching with the correct Microsoft terminology.

He already clarified that this is not a Web app, Ken. So neither the w3p (W2K3) process nor the aspnet (XP) process would come into play here.

It is hard to imagine having to run into this exception unless someone's using too many resources like MemoryStream's, SqlConnection's, and such. That's why (and also because of what Crusty pointed above) I asked for sample code.

 

[Markbnj]

I agree with Ken. This exception is unlikely, and the context and some sample code would really help to understand why it's happening. Crusty, you're very correct about the creation of temporary strings due to immutability, however in the normal case no reference is retained to all the intermediate forms, so the GC can be very aggressive about collecting them if memory gets tight.

 

[imported_Dhaval00]

Also, what version of the Framework are you using? 3.5?

 

[Red Squirrel]

I believe this app uses 2.0 but I do have 3.5 installed, so not sure if it just uses the latest or what.

Here is some sample code, note that there is lot of OOP involved but this is basically the order it gets executed as. There are about 1mil total entries in the DB, ranging from 10 to about 80k strings. There are only maybe like 100 or so that are 80k+ then it drasticly goes down towards the 10ks then lower.

First block, load data and create item instances:

//load Mobiles:
			Console.WriteLine("Loading Mobiles...");
			
			if(!Core.SQLCON.Query("SELECT * FROM mobiles;"))
			{
				Console.WriteLine("Error loading mobiles!  This is fatal and stuff.  Terminating");
				Console.ReadKey();
				return;
			}
			
			m_Mobiles = new Dictionary<Serial, Mobile>();
			
			
			
			while(Core.SQLCON.FetchRow())
			{
				int serial = Core.SQLCON.Reader.GetInt32(0);
				string serialstr = serial.ToString();
				int typeid = Core.SQLCON.Reader.GetInt32(1);
				string typestr = Core.SQLCON.Reader.GetString(2);
				string serializedata = Core.SQLCON.Reader.GetString(3);
				
				
				//ensure it has not been flagged as a bad type:
				
				bool badtype=false;
				foreach(string comp in m_badtypes)
				{
					if(comp==typestr)
					{
						badtype=true;
						break;
					}
				}
				
				if(badtype)continue;
				
				//get type:
				
				Type t = ScriptCompiler.FindTypeByFullName( typestr );
				if ( t == null )
				{
					FailLoadMsg(serialstr,typestr,"type not found");										
					m_badtypes.Add(typestr);
					continue;
				}
				
				//check for serialization constructor:
				
				ConstructorInfo ctor = t.GetConstructor( ctorTypes );

				if ( ctor == null )
				{
					FailLoadMsg(serialstr,typestr,"No serialization constructor");
					m_badtypes.Add(typestr);
				}
			
				//create object and add to world:
			
				Mobile m = null;

				try {
					ctorArgs[0] = ( Serial ) serial;
					m = ( Mobile ) ( ctor.Invoke( ctorArgs ) );
				} 
				catch 
				{
					FailLoadMsg(serialstr,typestr,"Could not construct object");
				}

				if ( m != null ) 
				{
					AddMobile( m );
				}
				
				
				PreLoadMobs.Add(new PreLoadMob(m,serializedata));
				
				
				mobileCount++;
			}

Second block: deserialization - take string, and populate the variables within the actual created item (the mobile in this case)

			Console.WriteLine("Deserializing mobiles...");
			
			foreach(PreLoadMob preload in PreLoadMobs)
			{
				preload.Deserialize();
			}

This is how a preloadmob looks like:

		private class PreLoadMob
		{
			Mobile Mob;
			string Data;			
			
			public PreLoadMob(Mobile pmob,string pdata)
			{
				Mob=pmob;
				Data=pdata;
			}
			
			public void Deserialize()
			{
				if(Core.DebugLoad)Console.WriteLine("------- Deserialize mobile {0} with data {1}...",Mob.Serial,Data);
				SqlReader reader = new SqlReader(Data,(int)Mob.Serial);
				Mob.Deserialize(reader);
			}		
		}

One issue I did just realize here is that I'm actually storing ALL strings into memory at the same time but while that is a lot of data, it's still not enough to go over the amount of ram I have. Durring this stage the app is using about 400MB of ram.

It usually crashes during the 2nd stage where it deserializes the string. (this just loops through by position to get the data in chunks).

Now one way I suppose that would maybe help is if I did not hold unto the data of every single string, and just had another SELECT statement for deserialize. I may look into that, though I think the cause of the crash is the fact of putting tons of data in a single string, it really does not care about all the other strings at that point considering there's tons of ram still left.

There are actually 3 loops of each, I just simplified it here. The initial load loops must happen before the deserialize.

 

[imported_Dhaval00]

I was able to reproduce the issue simply using string objects:

http://tinypic.com/view.php?pic=1fwzr7&s=5

It was being run on a Win2K8 Server VMWare image with 2GB of RAM. I ran it multiple times and the app would crash at around 500 MB on average.

Here is the code I used:

String test = "testing something";
for(;
{
test = test + test.Substring(0, test.Length/10);
Console.WriteLine(test.Length);
}


I also see a couple of other issues with your code (this is speculative based on the API you're using). I usually flinch when I see someone use SqlDataReader (again I am not sure if you're using it or not, but looks like you're). I'd recommend you switch to a SqlDataAdapter instead and do your tasks "offline." The SqlDataReader is a forward-only, always-connected object - puts extra burden on your application. I'd suggest you deserialize as well as process the strings using some sort of streaming or chunking logic.

 

[degibson]

Just for fun, I tried this in C++ on two linux platforms, too:
P4 RHEL5 2GB gcc 4.1.2
Clovertown RHEL5 24GB gcc 4.1.2

1 #include <iostream>
2 #include <string>
3
4 using namespace std;
5
6 int main() {
7 string s = "foo";
8 while(true) {
9 int len = s.length();
10 cout << len << endl << flush;
11 s = s + s.substr( len*9/10 );
12 }
13 return 0;
14 }

Both runs ended with this output:
255919150
terminate called after throwing an instance of 'std:ut_of_range'
what(): basic_string::substr
Abort

It looks like STL strings on this platform are suspiciously limited to ~256MB.

Edit: Silly emoticons

 

[Crusty]

Originally posted by: Dhaval00
I was able to reproduce the issue simply using string objects:

http://tinypic.com/view.php?pic=1fwzr7&s=5

It was being run on a Win2K8 Server VMWare image with 2GB of RAM. I ran it multiple times and the app would crash at around 500 MB on average.

Here is the code I used:

String test = "testing something";
for(;
{
test = test + test.Substring(0, test.Length/10);
Console.WriteLine(test.Length);
}


I also see a couple of other issues with your code (this is speculative based on the API you're using). I usually flinch when I see someone use SqlDataReader (again I am not sure if you're using it or not, but looks like you're). I'd recommend you switch to a SqlDataAdapter instead and do your tasks "offline." The SqlDataReader is a forward-only, always-connected object - puts extra burden on your application. I'd suggest you deserialize as well as process the strings using some sort of streaming or chunking logic.

I'm curious to know how long your string was when it crashed. Using a StringBuilder I was able to build a 137million length string before I got the exception, Task manager reported ~500MB usage.

 

[imported_Dhaval00]

Originally posted by: Crusty
I'm curious to know how long your string was when it crashed. Using a StringBuilder I was able to build a 137million length string before I got the exception, Task manager reported ~500MB usage.

If you look at the Console in the picture or while running the app, it actually shows the length of the string at the end of every iteration. In that screenshot, it was specifically 281,511,074. As a side note, up on reaching that limit, the application would come to a crawl before throwing the exception.

Edit:
I just "reflected" the algorithm for StringBuilder based on what Crusty posted. Looks like you won't be able to use a StringBuilder after all. StringBuilder actually uses a String to hold the data - however, where it differs is it will automatically expand the underlying array and copy data as it moves along. At this juncture (when it expands the array), the StringBuilder actually doubles the memory (this is also why I think your run failed at ~130M, Crusty). In essence, both String and StringBuilder will require contiguous chunks of memory.

At this point, you will have to use some sort of streaming logic... see if a StringReader/TextReader will help. I don't see how this fits your application especially given your serialization needs. As a foolproof solution, you should look at serializing your data as a byte array (even then, you'll need continuous chunks of memory when you *do* want to deserialize).

For debugging this, look up the MemoryFailPoint class on MSDN. Also, look at the CLR permon counters to see what's happening to the heap before your app poops. Regarding the memory bandwidth, I am thinking all of our tests have been limited by the 2GB process-limit on Win-32-bit OSs. Even then, the CLR will only allocate only so much memory to the AppDomain - which my guess at this point is 400-500MB. Overall, I think there is a serious design issue here somewhere in the application.

 

[Crusty]

Originally posted by: Dhaval00

Originally posted by: Crusty
I'm curious to know how long your string was when it crashed. Using a StringBuilder I was able to build a 137million length string before I got the exception, Task manager reported ~500MB usage.

If you look at the Console in the picture or while running the app, it actually shows the length of the string at the end of every iteration. In that screenshot, it was specifically 281,511,074. As a side note, up on reaching that limit, the application would come to a crawl before throwing the exception.

Lol oops... how could I have missed that link


After reading some of the OP's post in more detail, I would suggest altering how you are storing your data.

Why not use the built-in serialization of .NET to store your data in flat files and store references to the flat files in the database. You get the best of both worlds. If you want to store your actual data in the database then you should use more then one field for an entire objects data... that kinda defeats the purpose of using a relational DB.