Last week I’ve been assigned to solving problem with Out of Memory exception we were getting in our application. Oh, what I fine day it was! Something fun to solve, something useful to learn.
It’s been quite some time since I’ve bothered with memory usage in my applications. Usually it was so low, that no one cared and we concentrated on performance rather than memory print. Now things are bit different. We are not using all that much memory, but OoM exceptions were pretty common at some point. That was however solved by setting LargeAddressAware flag (http://msdn.microsoft.com/en-us/library/wz223b1z.aspx) to our executable – problem solved for most of our usages (and we are having a lot of free ram in our server boxes anyway so it’s not like we needed to minimize memory usage).
Then, few months later, problem re-appeared and this time it was persistent and repeatable. One report we were generating was failing. Unfortunately generating this report was taking 20+ minutes so not easy to reproduce. Still – I tried.
First I looked at task manager when in debug mode. Memory usage usually stayed around 500MB. Not too much, given that 32 bit Windows applications are given 2GB memory. Yet, few lines later, OoM appeared. Tried again – the same thing happened.
The code, that was throwing Out of Memory exception was Coherence.dll, .NET client for Coherence cache system. We weren’t having similar problems with Coherence before so I looked at what our code was doing. Nothing fancy really, trying to save some binary data – 61MB of data to be specific. Not huge amounts, but 3-4 times more than largest objects we were saving before. Funnily enough, saving those 61MB was fine – it was saving 4 character string in line after this that throws exception.
There are tools you want to use when you are having such problems. Profiles I mean. I thought there has to be something fishy going on. Ants Memory Profiler was tool I picked and tried to profile memory. It slowed down report generation to 40 minutes, but it profiled memory. Unfortunately it died when trying to snapshot memory usage just before OoM occurs. Well, I could’ve expected that. Looking through earlier memory snapshots did not reveal any obvious memory leaks or unexpected objects. All looked pretty much as expected. Still – exception was there.
So I decided to fall back to some lower level tools – VMMap (http://technet.microsoft.com/pl-pl/sysinternals/dd535533.aspx) – tool that will map your memory for given process, give you colorful diagram and show some details. Now – if you are going to use it – give it some time. It was processing my memory usage for 5 or 10 minutes before giving me results. So patience is the key.
However results are very useful, at least to me. Of course memory usage was low – 500 megs, like I said. What was not low was memory fragmentation. In Free type of memory there was something like 1400MB free but fragmented into close to 20 blocks, with largest being just a little over 100MB in size.
How is that helping me? Very much – given that Coherence.dll is serializing data into some kind of stream – MemoryStream probably. And MemoryStream has very interesting behavior. When it’s reaching its limit it tries to allocate new memory block (while preserving current one so data can be copied into new one of course). New memory block is going to be twice the current size. You have 1MB? It will go up to 2. You have 32? 64 it is. You see where I’m going with this? I have 64MB memory stream in memory (or something like that) already and while saving those 4 string characters I’m reaching its limit. It tries to get new one, 128MB (or something around this value) of continuous memory block (since MemoryStream is just byte underneath and arrays in C# occupy continuous blocks of memory). But as VMMap shows, largest block of memory is just 100MB!
Now, that clearly is Out of Memory situation. What can you do about this? Free some memory, remove some objects you no longer need. Or, as I did, limit saving data to something less. Since my data was one big xml underneath, zipping it made it small 5MB+ stream of bytes. Something that could easily fit into memory. Solution was implemented in 15 minutes and works perfectly.
What is left is – where does memory fragmentation come from? Well, obviously – you allocate memory, then you free some of it in the middle and here you go – you have small block of memory free, but not very usable if your other objects can’t fit into it. .NET platform is clever about it – it compacts memory after Garbage Collector runs (unless you pinned it, but that’s a different story). After compacting memory you should be left with one big block of free memory at the end and all allocated objects at the beggining of the memory. There is an exception to that rule however – Large Object Heap. If your objects are big enough (85k and above I think, but that’s probably not necessarily constant value), .NET will put your objects into separate heap and will not compact it since it would hit performance of GC hard.
Was that the case? Nope. LOH was just below 200kB, highly unlikely it would cause high memory fragmentation. What really was going on is, one of our component was working with COM objects. Those were not allocated by .NET framework, yet they reside in our process memory. So no compacting or anything. And we have quite a lot of those (probably around 200MB out of 500MB used by the application). Could that cause memory fragmentation? You bet it could! Didn’t touch that however, since impact of this change would be much bigger, change itself wouldn’t be as easy and we had other, better solution in place already (zip compression).