I now have DragonFlyBSD installed on a spare hard drive and I have verified that the VKERNEL feature works as documented. I will now examine the vmspace_xxx calls to see how I can implement a simple single-process LX binary executor which will run as userland code. This will allow me to quicky test my LX module loader code and to experiment on ways to simulate the 16bit APIs without requiring any 16bit code.
OS/2 2.0+ pretty much requires all the LDTs to be populated as 64KB aliases of the first 512MB of the memory map (8192 * 64KB) hence the 512MB limit. Some DLLs with the appropiate bit set in the LX header may be loaded above the 512MB mark but any memory allocated above there is inaccessible to the 16bit APIs. There is a DOSCALL available to do 16bit->32bit address translation and vice versa (DOSFLATTOSEL DOS32FLATTOSEL DOSSELTOFLAT DOS32SELTOFLAT) - and if people did actually use them then there wouldn't be a problem. Unfortunately most people just used the quick-and-dirty macros which IBM kindly provided like the SELTOFLAT macro in netcons,h
What I want to achieve with the userspace LX loader is to get demand page loading working. Unlike most binary loaders which map all the code into memory first, I want to only load and fixup code pages which are accessed. I will have to do basic checks at start to ensure that all external fixups can be satisfied and that there are no 16 bit segments but that is all. I will be relying on DragonFlyBSD's ability to trap page faults. Initially there will not be support for multi-threaded apps but then I may extend it with some form of thread simulation using multiple vmspaces for the threads: OS/2 needs this as each thread requires a different page mapped to the same logical address for it's per-thread data area - for reporting errors and for OS/2's thread-local storage. OS/2 only permits 64 words of thread-local storage which developers and compiler writers may use to point to an address for more space.
Phew - this was a long entry....

It appears that this blog has caught the attention of the DragonFlyBSD digest... Which is all fine and good except that this project is just a bunch of non-working code so far. Sorry guys, it's vapor right now!
Right now, I am thinking of how to simulate OS/2's legacy 16bit APIs without actually needing any 16bit code - although I don't intend to support any 16bit code segments, so all IOPL stuff won't work but things which used ioctls like Holger Veit's fastio$ may be supported. (isn't it silly that IBM never followed through and introduced the 32bit clean console APIs - MOU/KBD/VIO - from their aborted OS/2 PowerPC project but they did port GRADD).
Anyways, back on topic - for anyone reading this - there is no working code yet. Once I have something to show, I'll provide a link to a website where the code will be published. I plan to use a BSD style license.

I have been thinking... Now that DragonFlyBSD 1.8 was released today, if I use the MAP_VPAGETABLE facility, I have no immediate need for any in-kernel code. I could simply rely on that for an initial implementation.
Or, I can just leverage DragonFly's VKERNEL to accelerate development.