FC11 Memory leak to ruin your holidays

In the current world of cloud computing, cloud services are frequently updated. If your services are having small memory leaks, they will not even be detected; only big memory leaks could be noticed early enough before the next release kicks in.

But some organizations have holiday lock down policies which band updating cloud services shortly before or during holiday. When this happens, your cloud service processes get to live much longer, then small/slow memory leaks are going to add up and cause some slow down or damage to your cloud services.

A big source of memory leak is usage of System.Runtime.Caching.MemoryCache. This version of MemoryCache is implemented by a collection of Hashtable for data storage, and bunch of timers for cleanup. Normally MemoryCache works reasonably well. It’s often used by old code bases too much.

Singleton caches are not the problem because they’re designed to live forever. The problem are caches attached to something like a database client object. Such client object could be allocated, used for a while, then discarded. Then problem happens when such client objects are discarded. The MemoryCache it uses must be explicitly disposed. The reason is that the timer objects used by MenmoryCache are only disposed inside Dispose method implementation.

If Dispose method is not called, then the timer will live forever, holding the MemoryCache forever. This will eventually lead to high memory usage, high GC time, high CPU usage. The high CPU usage is coming from garbage collection, and timer queue firing. There is often heavy lock contention in timer queue.

So a slow MemoryCache leak could ruin your holiday, waking people mid of the night for troubleshooting, then fixing the problem.

Here is a simple repro:

This simple piece of code is just allocating 100 K instances of MemoryCache, putting nothing inside, with no objects holding those MemoryCache objects.

Normally, such objects without other objects holding them will be quickly cleaned up by garbage collection. But in this case, timer queue is holding them through timers. The situation is worse on machines with high core counts because the timer count is proportional to core count.

Try to run the code on your own machine.

Such problems can be easily detected from memory dumps or ETW traces.

Objects in memory after 100 K run (4.3 gb managed heap):

There are 100 K MemoryCache instances in memory, 800 Hashtable, 900 K timers, 24 million ExpireBucket. So each MemoryCache has N partitions implemented by Hashtable, each partition has 30 ExpiresBucket.

Here N is core count. MemoryCache would use many more timers and memory in high core count machines.