Slaying the Virtual Memory Monster (Dynamic-Link Library Relocation Edition)

I started something. There seemed to be a memory monster in my process and I wanted to find it.

 

To investigate who was pulling the blanket, I looked at the allocated virtual addresses with the wonderful windbg extension command "!address". However, this leads to information overflow and is generally of little use for this stage of my analysis.

But it indicates the problem could be rooted in the address space fragmentation caused by the loaded modules. I fed the output of "lm" into Excel to calculate the room between the load addresses. And there it was, my monster:

modules

Monster, thy name is MSCOMCTL.OCX, summoned by an innocuous legacy (i.e. VB6) library. It breaks apart my pristine address space into pieces. I wonder why that is, let’s dumpbin …

Dump of file C:Windowssyswow64mscomctl.ocx

PE signature found

File Type: DLL

FILE HEADER VALUES
             14C machine (x86)
               4 number of sections
        3F2F1329 time date stamp Tue Aug 05 04:15:05 2003
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
            2302 characteristics
                   Executable
                   32 bit word machine
                   Debug information stripped
                   DLL

OPTIONAL HEADER VALUES
             10B magic # (PE32)
            5.02 linker version
           ACA00 size of code
           56C00 size of initialized data
               0 size of uninitialized data
            89D8 entry point (275889D8)
            1000 base of code
           A7000 base of data
        27580000 image base (27580000 to 27685FFF)

Somebody disobeyed the ancient proposals of rebasing Win32 DLLs. Maybe because in 2003 no machine with more than 512MB of RAM would be running VB6 code?

"Optimizing" module load addresses was a great performance skill back in the days, when CPU cycles were scarce and jumping around in mapped memory hurt even more. Today, nobody could probably care less.

Since Microsoft owns and signs MSCOMCTL.OCX, I cannot do the old rebase trick and let the app modify the file itself, which would be utterly stupid anyway. But, there is no version of LoadLibrary, that would allow me to "unforce" the image base address.

 

So what can I do?

 

SYMPTOM:

Premature fragmentation of the address space.

CAUSE:

Some modules are linked with arbitrary, inappropriate load addresses.

RESOLUTION:

Block the respective address ranges as early as possible, but only temporarily using VirtualAlloc(…,…,MEM_RESERVE, …) and VirtualFree(…, MEM_RELEASE). This works only for modules that are loaded late in the game, most likely through delay-loading, JIT compilation or COM activation.

 

Let’s see where this will leaves us …

Advertisements
This entry was posted in Coding Horror. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s