Summary:
The ffcall library which is used in the GNUStep base library has a problem on Windows XP SP2 (and later) if Data Execution Prevention is turned on for all programs.
Details:
Here is a patch to ffcall 1.10, that allows the trampoline code to work with Windows Data Execution Prevention.
What is Data Execution Prevention?
In Windows Server 2003 & Windows XP SP2, Microsoft added a feature (if you have the hardware to support it), which will prevent the execution of code in areas of memory marked for data. So specifically this protects against a class of buffer overrun attacks.
I know you are saying, “So what, I don’t have self modifying code” that is what I thought too. So go and change your settings and turn on DEP. Go do it now, I’ll wait here…..
Ok, perhaps your program still runs fine, but mine didn’t. Turns out that the GNUStep implementation of NSInvocation uses the ffcall library to make the dynamic function calls. The ffcall implementaion of trampolines (basically a function pointer with context) allocates some memory and writes some opcodes in to it to set up the context state then JMP to the real function. Oops, that is self modifying code and doesn’t work any more.
The Solution: Windows doesn’t actually ban all execution in writable pages, just in data pages. Most unix OSes have similar limitations. We just need to tell Windows that we’d like a writable data page to put our code into. This is done with the VirtualAlloc function. For example:
void *addr = VirtualAlloc(NULL, bytesneeded, MEM_COMMIT, PAGE_EXECUTE_READWRITE)
Now, VirtualAlloc can only allocate multiples of the pagesize, and the bytesneeded in that call will be rounded up. So don’t be using it like malloc and calling VirtualAlloc over and over again with tiny little values. The default page size is 4k on workstation and either 4k or 2M (yes 2 megabytes!) on server platforms.