Making OS/2 family programs with Visual C++

18 Sep 2023

Back in the day, I loved OS/2. Unfortunately developing for it was difficult due to lack of tools and documentation - in the pre-Internet era, finding books on the Windows API was a lot easier than the OS/2 API. Recently I've also been wanting to run some of my tools on DOS. So with an abundance of optimism, I tried to kill two birds with one stone: write programs for the OS/2 Family API that can run on DOS but also OS/2, while learning a bit about the OS/2 API.

The Family API tooling consists of two parts. First, an API.LIB which is a statically linked piece of code providing implementations of OS/2 functions on DOS. Many of these translations are straightforward. Second, a BIND.EXE program which takes an OS/2 executable, adds a stub MS-DOS program that implements an NE loader, and links the code from API.LIB so the NE loader can resolve OS/2 functions to the DOS translation.

Unlike WLO, both of these components were widely distributed, with C 5.1, C 6.0, MASM 5.1 and MASM 6.0. There may have been other tools distributing these.

16 bit development tools for OS/2: MASM 6.0b and Visual C++ 1.5

For tools, I got a copy of MASM 6.0b from eBay. This had some unexpected good points and unexpected bad points. In hindsight these were all obvious, but they hadn't occurred to me earlier.

The good:

The bad:

Nonetheless, it did have the core capabilities I was looking for: the ability to write 16 bit OS/2 programs and bind those into DOS.

Very quickly I found that the best development environment for writing these programs is Windows NT. That was unexpected, but NT can run both DOS and OS/2 programs in a unified console, while also supporting Win32 development tools. Both the DOS and OS/2 debuggers work on NT, and can even support 80x50 mode. NT was also a good choice due to using Visual C++ 1.5 as a compiler.

Visual C++ 1.5 has some good and bad points as an OS/2 compiler:

Unlike Win32, the startup code in OS/2 depends on assembly. A newly launched program is informed of its state in x86 registers. For a non-trivial program, the startup code needs to save these registers before launching C. These registers inform the program of things like command line arguments, which is needed before getting to main.


.MODEL large, pascal, FARSTACK, OS_OS2

__startup PROTO FAR PASCAL

.DATA
public __acrtused
__acrtused = 1234h
_EnvSelector    WORD ?
_CmdlineOffset  WORD ?
_DataSegSize    WORD ?

.STACK 6144

.CODE
.STARTUP
; OS/2 Arguments
; AX Selector of environment
; BX Command line offset within environment selector
; CX Size of data segment

mov [_EnvSelector], AX
mov [_CmdlineOffset], BX
mov [_DataSegSize], CX

call __startup

With that, a C function called "__startup" can resume execution. Note at this point it has no argc or argv, just a stack, and access to the OS/2 API.

Next, writing the rest of the C runtime.