Why?
There are many people now who know unix, like unix, love unix even, but for whatever reason they run Windows on their PCs. It may be because we are forced to use Windows by our employer or maybe there is a vital application that is only available for Windows and for which there is no unix equivalent, but whatever, we still wish we could:-
Use a good shell.
Run together a pipe of commands like find and grep.
Use a powerful text editor like vi or emacs.
Setup background tasks to run constantly or at set intervals.
Accept network connections and authenticate user logins to interactively execute programs.
For sure, there are Windows equivalents of the great majority of unix software. Often though the Windows versions are not as powerful, do not integrate as well, and probably most importantly, they are not free. By 'free' we are talking about the availability of source-code as well as the fact that the software in free in financial terms.
The remainder of this document describes the three main kinds of interaction possible between the Unix and Windows worlds. Then I describe my plan for Implementing a complete Unix on Windows solution.
Four levels of integration
In my view there are four levels of integration which are described below.
1. Lower forms of integration
These include file sharing via NFS and Samba and Dual booting machines. Dual booting simply allows a single machine to either load and run Windows or Linux. That is however as far as the integration goes. Depending on which flavour of UNIX you boot into, you may be able to read and write to the parts of the disc that are storing Windows and it's files when you boot into UNIX; It is unlikely that you will be able to read and write the UNIX filesystems from Windows though.
Network file-sharing via NFS and Samba are altogether more interesting. In the old days, a multi-user machine (typically running UNIX or VMS) was regarded as a multi-user machine because many people could simultaneously log into it and run programs. Since the personal computer in all its forms became ubiquitous running software on shared computers has become much less important. The principle use for server-class machines now is file and print serving.
It is reckoned that the vast majority of servers running Windows NT Server are used simply for file and print services. The majority of the remainder only run one or two extra services such as email or web page serving. This presents an interesting scenario: Linux is arguably more stable than NT; it is certainly less resource-intensive; it runs file and print services using Samba at least as well as NT Server does. Most important of all though is that you don't have to pay Microsoft a big fat license to run it - no copy of NT Server, no Client Access Licenses.
Interesting as these techniques are, they do not fulfil the requirements for Windows/UNIX integration set out at the top of this article.
2. Porting the application
Arguably the easiest way to get a Unix application to run under windows is to port it across. That is: copy the source-code onto a windows machine which has the necessary development tools installed; Try to compile the application; Fix the many thousands of problems you come across as you go.
Downsides of the application porting approach are:
It is a lot of work because the Unix APIs (commonly referred to by the name of the 'official Unix standard POSIX) and the Win32 APIs differ significantly. The level of difficulty is obviously proportional to the size of the program you are trying to port as well as the amount of API calling done by the program. For example: gzip would be relatively trivial to port; screen would be a nightmare; gcc would somewhere in between because although it is more complex than screen, it interacts less with the operating system.
Unless you are very careful about the way you modify the source of the original program, you will find it difficult to keep the Unix and Windows versions of the program in step. The author of the Unix version will make some changes to fix a bug or to add a new feature and you will find yourself having to re-port whole sections of the program resulting in lots of wasted effort. The way to prevent this from happening is to co-ordinate your porting effort with the original author of the software (making it an 'official' port). The source will then be separated out into three sections: Unix, Windows and common. Most changes will only affect the common code and so a simple re-compilation on the Windows side is all that is required when the author makes changes.
The downsides of the application porting approach can in most cases be overcome with some effort. However, this effort is usually considerable - especially in the case of graphical applications.
3. An API emulation library
This is typified by the CygWin32 package from Cygnus Solutions, although there are other more expensive commercial solutions available such as NuTCracker.
The purpose of an API emulation library in this context is to allow you to compile a UNIX utility/application on Windows with the absolute minimum of changes. Constructing such a library is somewhat more complex a task that it at first seems.
Most modern operating systems owe a lot of their design to UNIX. Even humble old MS-DOS has some important concepts taken from UNIX. Version 1 of MS-DOS was pretty much a clone of CP/M. From version 2 onwards though almost everything about the API changed and file handles and streams (UNIX-style) were introduced.
The win32 API has a large proportion of the calls a UNIX programmer would expect to see. However, many of them behave slightly differently. In addition there are things that a UNIX program will do that Windows programs in general do not do and hence no facility exists on Windows to do them. For example, most common UNIX programs expect to be connected to a terminal of some description. The best Windows can offer by itself is if you happened to load ansi.sys in your command window.
Small mismatches in the way certain API calls work can be patched by an API emulation library without too much difficulty (but great tedium). When there are facilities missing wholesale from Windows, some can be emulated in the library, but others require the porting of large UNIX sub-systems to the Windows environment. This is the crux of the problem with this approach: UNIX application expect not only the UNIX API to be available, but also the many other applications, utilities and daemons that make up UNIX environment. You would have to port each and every item across from UNIX in order to get some confidence that a UNIX application would run on Windows without non-trivial modification.
Despite what I've said in the above paragraph, the people at Cygnus have done a fantastic job with the Cygwin32 library. Judging by comments posted to the Porting Page, many popular UNIX applications and utilities will compile and run on Windows with very few and often no porting effort required. Impressive!
4. Unix sub-system
Emulating the UNIX environment on Windows simply by wrapping the application code in a complex library is in my view a sub-optimal solution. Ideally applications should not require any modifications and better still, binaries from popular UNIX-like OSs should run unmodified subject to processor type.
A UNIX sub-system approach attempts to achieve this by emulating not only the UNIX API, but the whole UNIX environment. That means you not only get to call open(), but you get to call it on /dev/tty. You can also find out the status of processes running by examining the data under /proc and you gain the concept of users being logged in by means of /bin/login and /var/utmp.
UNIX kernels tend to be large and monolithic. This is gradually changing. The most advanced commercial UNIX variants - AIX and Solaris are highly componentised. Linux has it's modules which make it easier to change hardware configurations and test new kernel code. However, in all these systems there is a lot of work done by the kernel that is not what I would call 'core'.
Core kernel code includes: memory management; processes creation and management and the loading of those parts of the OS which are not core. There are many parts of a traditional UNIX kernel that are not core: device drivers; filesystems; virtual terminal handling; network protocol stack, etc.
A UNIX sub-system for Windows would utilise the existing process and memory management in Windows. It would also use all the device interfacing provided by Windows. The UNIX binary should be loaded into an ordinary Windows process and system calls intercepted and matched to Windows system calls. This is not all though. A UNIX-like filesystem should be provided to the process. It should be able to create files with any characters (except '/') in their names. It should be able to set the full range of permissions on files and even make them setuid. While we are on the subject of setuid, the UNIX sub-system should give the concept of user-id and effective user-id and the same for group ids etc. The concept of file ownership should be introduced to the filesystem in the emulation layer.
Although all these things could be provided by the API emulation library, it would be far simpler to provide them in a separate sub-system. There are also things that cannot be provided by a library. When UNIX processes start other processes they do so in a two stage procedure: First they call fork() which creates a almost identical duplicate process to the current one called the 'child'; If the call to fork() is successful the child process calls exec() which then obliterates the program and replaces it with a new one. On Win32 there is a single CreateProcess() function that creates a fresh process with a new program to run from the start. There is no way to clone an existing process the way fork() does on UNIX. As I said, most UNIX programs call exec() immediately after forking, but this behaviour cannot be relied upon. Indeed some very important applications (web and ftp servers for instance) carry on running in their current state after forking.
My Plan
A. Filesystem
It is plain to see that even with the long filename support introduced in Windows 95 and NT4, it is not possible to replicate the same level of information on each file in the filesystem that UNIX needs. There are many characters that are not allowed in filenames on Windows (UNIX systems typically allow everything except a forward slash).
This could be solved using an extra 'properties' file in each directory to store the extra information required like the RockRidge Extensions used on UNIX CD-ROMs. Disadvantages to this approach include:-
Performance loss from having to read and update the properties file every time a file in the directory is touched
Windows programs looking in the directory would see the mangled filenames and the properties file which would not be helpful to them (remember our goal here is integration - if all we wanted was to run UNIX programs we would just install a UNIX on our PC and be done with it).
It does not solve all our problems.
It does not solve all the problems because there is one thing that nearly all UNIX filesystems have that the FAT filesystem used on Windows 95 and some NT installations does not have: Hard Links. On UNIX a file is not stored in a directory as you might think. Every file is uniquely identified by an inode number allocated to it by the operating system when it is created. The directory entries that the user sees are merely named 'pointers' to the inodes. Most UNIX filesystems will allow you to create more than one link to a particular inode thus magically causing the same file to appear in more than one place (possibly with different names) at the same time. NTFS on NT does support 'hard linking' of files so they can appear in more than one place at a time, but it does not have the same more subtle distinction between the directory entries and the files themselves.
What is needed is a complete implementation of a UNIX-grade filesystem within Windows. Most people will not want to re-partition their drives the way dedicated Linux users do, so ideally it would sit within an existing FAT or NTFS partition. It is possible for a program to allocate a fixed contiguous region of a FAT or NTFS partition for it's own exclusive use (look at the way the pagefile works on both Windows 95 and NT). Once this area has been allocated Windows will not touch it. It is then up to the UNIX sub-system to implement a UNIX filesystem within it. Code to implement high-quality extent-based filesystems is freely available thanks to the many free unices developed by the net community. The UNIX sub-system can also then install a network re-directory to expose this filesystem to Windows applications in the same way than an external Linux machine might expose it's filesystem across the network using Samba.
B. Processes
Each process within the UNIX sub-system should be implemented as an ordinary process within Windows. The UNIX sub-system will keep track of process attributes that Windows doesn't like their user, group, registered signal handlers etc.
It should be possible to load binaries produced on real UNIX systems like Linux and FreeBSD. All that is required is a routine to read their file format and load the segments (code, static data, etc.) into the relevant places in the Windows memory map. UNIX style system calls (usually implemented using traps) will need to be patched up so they call the UNIX sub-system instead.
One essential is that an equivalent of fork() be provided. Cygwin (mentioned in the section on API Emulation Libraries) emulates fork() by creating a new process that copies all the memory of the old process into it at start up. This takes an appreciable length of time even for simple programs. For instance, when you type commands at the shell prompt on a UNIX system for all but the simplest commands a new process is started. With a memory-copying implementation of fork() the whole of the memory consumed by the shell (which can be large for a modern shell like GNU's Bash) has to be copied into the new process only for it to then overwrite it.
If program executables are loaded into memory using memory-mapped files (as they are in Windows and most unices) and all memory allocation is done using memory-mapped files (this is rather more unusual) then when a new process is started, instead of copying the memory from it's parent it could just map the same memory instead. Under Win32 memory-mapped files can be mapped as read-only (for the program text) and read-write memory (the heap and stack) can be made to copy-on-write when it is mapped into the address space of more than one process.
There is a slight problem here though. If you read the descriptions of the necessary memory-mapping calls in the Platform SDK you will see that Windows 95 cannot map a file into a specific address in one process and then map a different file into that same address in another process. NT has no such restriction. This makes (on 95 at least) the implementation of fork() somewhat more difficult. We can still use the technique described in the above paragraph, but we have to use an executable format that allows use to re-locate the executable when it is loaded. We then make sure we map everything at different addresses allowing processes to share pages of memory as and when required. This is a sub-optimal solution to the problem, but it may be the best we can do on Windows 95 - certainly it is the best we can do using the Win32 API. It may be possible to implement a true fork() on Windows 95 by writing a VxD service. VxDs can interfere with the memory system in Windows at the very lowest level (the page tables) and can install hooks into places most Windows programmers don't even realise exist. This will take some time to develop though as I have little knowledge of VxDs at the time of writing.
C. Other UNIX-isms
Applications running under our UNIX sub-system will expect to write their output to /dev/tty (actually they will mostly just use stdout, but a typical UNIX libc will just connect that to /dev/tty). Thus connecting /dev/tty is just one of the things we will have to do before applications can be convinced that we are running UNIX rather than Windows.
Wait a minute though! What do we mean when we say "console"? Windows 95 has the MSDOS window and NT has the Command Prompt. These will both do a passable ANSI terminal emulation, but it is by no means what a UNIX app will expect. The UNIX app will in most cases want to read a termcap entry for the type of terminal named in the TERM environment variable. To give character-based UNIX applications what they expect they need to be run from a good quality terminal emulator. They need to be able to find out what type of terminal is being emulated and they need access to a valid termcap entry for that type of terminal.
One solution that would avoid writing a lot of new code would be to only allow user access to our UNIX sub-system through the network. This may sound strange at first, but it makes a lot of sense. Our UNIX sub-system can be developed and managed a lot more easily if it is invisible to the outside world - it has no user-interface. If we want to run programs on another UNIX machine we typically would log into it using a telnet client across a network of some description. Unlike NT, with UNIX systems the remote user can (subject to security restrictions) do anything that a local user can - there are no 'second-class citizens'. You can connect to your own machine through the network, so why no make the only interface to the UNIX sub-system a network one? This goes for graphical applications too - X11 was designed to communicate with applications through a network layer.
So once we have the ability to store files on disk in a UNIX-like fashion and execute processes, all we need to complete our system is a means for the user to get in and do useful work. To start with any Windows telnet client will do the job. When we get more advanced we will need (or at least desire) a Windows X11 server [client and server are back-to-front in X11 talk remember?]. There are several commercial X11 solutions for Windows and at least one free one, but there is no good free X-server available at the moment. That will be the subject of further investigation from Exaflop.org.
Conclusion
It is the view of the author that the ability to maintain a UNIX environment on a Windows-based PC is a desirable goal, if that environment is sufficient to compile common UNIX software without major modification, and run that software at a similar speed to a real UNIX OS on the same hardware.
It is not on desirable, but also achievable. This however is not to say that it would not require a lot of work. The lack of a true fork() function call presents one of the biggest problems, but even this can be overcome with some effort - even on Windows 95.
