I started something. There seemed to be a memory monster in my process and I wanted to find it.
To investigate who was pulling the blanket, I looked at the allocated virtual addresses with the wonderful windbg extension command "!address". However, this leads to information overflow and is generally of little use for this stage of my analysis.
But it indicates the problem could be rooted in the address space fragmentation caused by the loaded modules. I fed the output of "lm" into Excel to calculate the room between the load addresses. And there it was, my monster:
Monster, thy name is MSCOMCTL.OCX, summoned by an innocuous legacy (i.e. VB6) library. It breaks apart my pristine address space into pieces. I wonder why that is, let’s dumpbin …
Dump of file C:Windowssyswow64mscomctl.ocx PE signature found File Type: DLL FILE HEADER VALUES 14C machine (x86) 4 number of sections 3F2F1329 time date stamp Tue Aug 05 04:15:05 2003 0 file pointer to symbol table 0 number of symbols E0 size of optional header 2302 characteristics Executable 32 bit word machine Debug information stripped DLL OPTIONAL HEADER VALUES 10B magic # (PE32) 5.02 linker version ACA00 size of code 56C00 size of initialized data 0 size of uninitialized data 89D8 entry point (275889D8) 1000 base of code A7000 base of data 27580000 image base (27580000 to 27685FFF)
Somebody disobeyed the ancient proposals of rebasing Win32 DLLs. Maybe because in 2003 no machine with more than 512MB of RAM would be running VB6 code?
"Optimizing" module load addresses was a great performance skill back in the days, when CPU cycles were scarce and jumping around in mapped memory hurt even more. Today, nobody could probably care less.
Since Microsoft owns and signs MSCOMCTL.OCX, I cannot do the old rebase trick and let the app modify the file itself, which would be utterly stupid anyway. But, there is no version of LoadLibrary, that would allow me to "unforce" the image base address.
So what can I do?
Premature fragmentation of the address space.
Some modules are linked with arbitrary, inappropriate load addresses.
Block the respective address ranges as early as possible, but only temporarily using VirtualAlloc(…,…,MEM_RESERVE, …) and VirtualFree(…, MEM_RELEASE). This works only for modules that are loaded late in the game, most likely through delay-loading, JIT compilation or COM activation.
Let’s see where this will leaves us …