This is one of many stories where your machine is high on CPU and RAM and disk but low on motivation to do it’s job.
"In the test lab, some test work items failed sporadically."
It’s those bugs that make you loose sleep beyond compare. Especially when you see in the trace files and memory dumps that the offending method is CoCreateInstance and the error code is 0x80070583 (HRESULT_FROM_WIN32(ERROR_CLASS_DOES_NOT_EXIST)), but the class factory or object constructor is never even entered.
What’s worse, the problem only occurs on Windows XP SP2 and neither on Windows XP SP1 nor Windows 2000 SP4 nor Windows 2003 SP1 and only after 2 to 12 hours of intense load.
Eventually, you figure out it’s about the COM security configuration – using Interactive User everything runs fine. But that’s not what COM servers are built to run under.
So, with a little test you find out that on Windows XP SP2 you can instantiate roughly 50 out-of-process COM servers under a specified service user account. And then things fall apart, ususally with 0x80080005 (CO_E_SERVER_EXEC_FAILURE).
OK, 50 COM servers per machine seems to be quite sufficient for anything run on a workstation, doesn’t it?
Well, actually not. Our software needs to control a specified number of scientific instruments per machine and we’re gathering lots of data. That data is precious enough to our customers to make us go great lengths to isolate everything in processes to be as resilient as possible in processing and persisting it.
Using Microsoft PSS we were able to track the problem down to the Desktop Heap of the non-interactive WindowStation.
The documentation in this area is, well, sophisticated.
All non-interactive services share a WindowStation. The Desktop Heap in that WindowStation is limited. This limit is hit hard under Windows XP SP2. The other systems are smarter about it, it seems.
The natural instinct is to increase the desktop heap of the non-interactive WindowStation and, hey, there is a registry entry for that.
Unfortunately, the registry key in question is very, very vital to Windows. So if it is messed up, you cannot run Win32 programs on your system anymore after reboot. Try to get around that one! (Hint: Use a second system to mount and fix the messed up registry.)
That’s why I won’t explain the necessary change directly but link to the Knowledge Base entries.
PRB: User32.dll or Kernel32.dll Fails to Initialize
"Out of Memory" Error Message Appears When You Have a Large Number of Programs Running
The default is 512 and for us, a value of 2048 for the non-interactive desktop heap appeared to be sufficient, but higher values have been suggested.
However, I would not set the number any higher, because as far as I understand the documentation, all desktop heaps have to fit in a 48MB fixed buffer, although there is not evidence that this is correct.
Under Windows Vista, memory management will be different and the problem is likely to be addressed. Search for Vista Desktop Heap to find a presentation on this topic.