CAFTP Editor’s Note: My temporary second job will be ending soon, so I’ll be able to start posting more often. Until then, enjoy this interview about MS OS/2. Just think, if Microsoft had not fallen out with IBM, we’d be on OS/11 by now. Maybe.
From the May 1987 issue of Microsoft Systems Journal
OS/2: Turning Off the Car to Change Gears
An Interview with Gordon Letwin
by Lori Valigra
The development of Microsoft’s multitasking operating system has taken many twists and turns since Microsoft began working on it three years ago. The many stages it has gone through so far have resulted from working with the schizophrenic Intel 80286 chip, which can run in either of two incompatible modes: protected (286) mode, and unprotected (8086 or “real”) mode. The main problem has been to get an operating system which runs both protected- and realmode programs. Microsoft solved the problem with mode switching—a feature which essentially works by mimicking the act of turning off the computer and restarting it, similar to turning off a car in order to switch gears.
The new operating system began as an ambitious project to produce an operating system for Intel 80286-based computers running a mixture of office automation applications. This new operating system was to be capable of multitasking and compatible with the protected 286 mode. But as the project got too unwieldly, Microsoft split it up in early 1985, announcing a version called DOS 4. This version incorporates multitasking but does not include protected mode features. The DOS 5 project, as it was called, is intended to support the protected mode and add other features.
OS/2, as the project has finally been named, was easily the most difficult and challenging program written at Microsoft to date, with more than 350,000 lines of code. While Microsoft generally feels that the most efficient projects are those staffed by one person, the complexity and time constraints of the OS/2 project required that the company assign more than 35 programmers to it. Managing the project—dividing the programmers into teams, dividing the project among them, and coordinating the various teams’ modifications to the program’s code—became an enormous challenge. Adding to the challenge was the joint development of the product with IBM, a company whose culture and methodology are quite different from Microsoft’s.
Microsoft Systems Journal talked with the project’s chief architect, Gordon Letwin, to learn more about the planning, development, and cooperative effort that went into the development of OS/2.
MSJ: What was the goal of the OS/2 project?
Letwin: We tried to create the underpinnings for the ideal office automation operating system. In our opinion, that operating system would allow applications direct access to high-bandwidth peripherals, offer device-independent graphics drivers, provide multitasking without performance degradation (compared with a singletasking system), run programs in both the protected 286 and real 8086 modes of the 286 processor, and provide a protected environment to assure system stability. I think we’ve done a pretty good job.
MSJ: Microsoft has been developing multitasking operating systems for three years now, beginning with DOS 4. What was the purpose of that operating system, and where does it stand now that OS/2 is out?
Letwin: DOS 4 was the first product to result from Microsoft’s multitasking DOS effort. We began it even before IBM introduced the PC AT. It was an ambitious product that was originally to include a protected mode with mode switching abilities so it could run on the 8086 or the 286.
A general-purpose multitasking system needs to run in both modes: the unprotected 8086 mode so that we can run existing DOS applications, and the protected 286 mode so that we can multitask arbitrary, unrelated applications. But the architecture of the 286 caused some delays. Although we knew the project would be difficult, it was only after we’d gotten deeply into it that we realized just how difficult it would be.
As a result, DOS 4 became too complicated for our schedules. Because of the pressure of customer demand as well as that of previous commitments, we broke the project into two parts. DOS 4 runs only in real mode and provides multitasking only for specialized applications. DOS 5, which has now been released as OS/2, includes the protected mode and other features. DOS 4 was delivered in the last half of 1986 and is being sold in special application environments, primarily in Europe. It is a specialized product that can share the market with OS/2, because it runs on 8086 hardware, while OS/2 requires a 286. The move from DOS 4 to OS/2 was a gradual evolutionary process. OS/2 is by far the most complex and sophisticated operating system project we’ve worked on. To offer multitasking in a protected environment, we developed it jointly with IBM to get around the constraints of the 286.
MSJ: How does OS/2 compare with earlier multitasking operating systems like Concurrent CP/M?
Letwin: Unlike OS/2, Concurrent CP/M has a major problem. It serializes access to the file system. When multitasking you need to make the operating system features such as the file system accessible to multiple programs at a time. But Concurrent CP/M didn’t do this. It only allowed one program to call the file system at a time. So in a real life situation with lots of I/O, Concurrent CP/M was too slow.
MSJ: What are some of the features of OS/2 that you consider particularly neat?
Letwin: One is the seamless system service interface, which means that system services don’t need to be in the kernel. Some can be provided by subroutine libraries and others by system processes. Many services can be upgraded piecemeal, in the field, without changing the whole operating system. This is accomplished by a technique called “dynamic linking,” which allows external procedure calls to be linked up at execution time, rather than link time. This same mechanism is used as the standard system interface between applications, utility packages, etc.
Another exceptional feature is direct hardware access, which allows an application to write directly to the hardware. For example, it will do special graphics on a screen.
MSJ: How did you get around the limitations of the 286 chip?
Letwin: The 286 is designed to run in only one mode, either real mode or protected mode. Since we knew we needed to support programs written using protected mode, a group of us, including Bill Gates and some project engineers, brainstormed to try to figure out how to run 8086 mode applications from within protected 286 mode. One option was “real mode emulation,” where we could emulate the operation of 8086 real mode by doing a lot of special things to the protected mode segment tables. IBM _ was examining this technique independently. We thought emulation would be too slow, since the 286 would run at one-third the speed of an 8086. IBM’s research later showed this to be the case. Soon thereafter, Intel began shipping a chip with a bug in it that would stop the technique anyway, so we tried another approach: mode switching.
The 286 chip switches from real mode to protected mode very easily, but it was not designed to switch back from protected mode to real mode at all. You could only switch back to real mode by causing a full system reset. The idea of doing a full system reset while running a multitasking system was very radical. IBM was aware of the mode switching technique itself and had the capability for it in the AT BIOS for use during power-up diagnostics and for special block copies from high memory. However, this feature was never intended for use during full system operation. What we were thinking of was similar to switching gears in a car by turning the motor off. The idea was to master-clear the processor and restart it. The question was, would this interfere with DMA transfers that might be happening at the same time, lose pending interrupts, or fail in some other way? The IBM ROM code was designed to “switch gears by turning off the car” with the car standing still. We wanted to switch gears with the car traveling at highway speeds, and without causing a jolt. Working together with IBM, we set up an experiment to see if it would work, and it did. However, it took us months to convince everyone involved that the technique was feasible. The first time we presented the idea to IBM Boca, they gave us a funny look and suggested that perhaps we’d had a bit too much sun.
There is one difficulty with mode switching, however. Because the designers thought that it would be an infrequent operation, done primarily during the power-on diagnostics, they implemented it in a fashion that saved some components but produced a very slow mode switch: typically 1,000 microseconds, round trip. During this | millisecond interval the system cannot respond to interrupts. Any interrupts that arrive then are processed after the mode switch. This delay would normally be a problem only for high-speed communications at 96K bps or faster while you are simultaneously interacting with a real-mode program. You may experience data overruns while the realmode screen group is on the display. Mode switching will affect any program that requires a very high-speed interrupt service rate faster than 500 Hz. If the communications protocols on your system can handle missing characters by asking for a retransmit, and most can, then communications will be okay.
MSJ: Why is protection so important in OS/2?
Letwin: We knew we needed a multitasking system to accomplish our goals, and we knew we needed to have it protected. If you run a multiprocessor system, software can accidentally read or change other programs or data if the memory is unprotected. You can’t stop it and you can’t tell it happened. The system can crash or you could corrupt data. Even in an unprotected DOS 3 system you can have problems with running programs simultaneously. SideKick, for example, loads into memory and terminates itself. It keeps memory in use, but the system doesn’t know what is in that memory location. If you load several such programs at the same time and they conflict on how they use memory or interrupt vectors, the software will not work correctly, or the system will crash. So in an unprotected environment the system is always vulnerable to flaky software or conflicts between software products from different vendors.
Of even greater importance than this is the fact that we needed to use protected mode to allow programs to access more than 640K bytes of memory. The 640K limit in earlier versions of DOS is a result of the addressing limitations of real mode. Protected mode programs are released from this restriction, and can use many megabytes of memory. As memory costs continue to drop and memory consumption by applications rises, it has become unacceptable to be restricted to the 640K that real mode provides.
MSJ: How did you plan such an ambitious project, both conceptually and in getting the needed resources?
Letwin: The first step was to decide what we wanted the system to do. Then, when we discovered how many of those capabilities we could implement, and by when, we could revise our goals. As a result, the initial planning process was iterative. We needed to have multitasking and to execute existing DOS 3 programs. While there were still only a few engineers on the project, we thought about techniques such as mode switching and when necessary, prototyped some code. As we got a better handle on what we could do, and how long it would take, we met again with Bill Gates and revised our goals. The tone of the first phase was as ambitious as possible, and we listed all the things we thought we could do. The second phase involved determining which of those features were feasible in the required time frame. As soon as the general scope of the project was determined, the engineers met in groups of two and three to lay out further details of the architecture of the system.
After the initial meetings came the joint design work. Engineering then finalized the goals. Because of the scope of the project, this design phase took a few months.
Next was the sign-up phase, a tradition in the computer industry, where we explained the excitement, the importance, and the tight schedule of the project, then let everyone “sign up” for a long stretch of hard, focused work. You can’t force someone to excel; you’ve got to provide the opportunity and the encouragement and then let them excel.
Then the actual work began. We decided on the key systems and the key characteristics of the product. Next came the design sessions for each of the key systems. About a half dozen people were involved, typically three from Microsoft and three from IBM.
During the coding months, my job was to understand fully all of the parts of the system and to make sure that everything worked toward achieving our overall system goals. In a project this large, with this many people working on it, it’s easy for design elements to diverge. One project member might think that speed is more important than code size, and write his code that way. Another might think the opposite and write his code the other way. The result would be that neither goal is achieved. I participated in the design of each component to insure that the goals and design philosophies of the system’s architecture were consistent throughout the code. On most projects, this role can be fulfilled on a “part time” basis, leaving time for the chief designer to also get to write some code. Unfortunately, on the OS/2 project this position required a full time effort, leaving me little opportunity to write any real code.
MSJ: How did you divide up tasks on the project?
Letwin: The most efficient programming is done by one person. It’s a classic saying in the industry that “if one person can do it in one year, two people can do it in two years.” But in a project like OS/2, although one person might be able to do it in 8 man years, you can’t afford to wait 8 years, so you have to make it a multi-person project and have it take 30 man years. In order to reduce the inefficiency introduced by having multiple people on a project, it’s important to break the work up properly, and to have each piece of work be a one man project. This project had about 35 people on it.
The programmers worked on individual parts of the code, but they were organized into teams with a maximum of eight people each to facilitate coordinating their work. There was a test team, two teams of developers each headed by a senior engineer, a build team with four people, a test team with five people, and myself as the chief architect. This project was so complicated, especially since development work was simultaneously taking place at IBM in Boca Raton, Florida, that we needed a build team just to coordinate the sources, build the executable binaries, and otherwise bring organization to the physical aspects of software development. The test team, as its name implies, was responsible for the development of test cases, bug tracking, and performance measurement.
MSJ: What was the most difficult part of managing the project?
Letwin: This project’s very large staff added to the complexity of integrating changes made at the IBM locations and timing the development between the two sites. We had to be especially careful that any changes made didn’t affect other parts of the project. In the source code control system, for example, we split the source into about 200 modules, with modification tracking on each. We had a rule that you couldn’t add a change to a module until you’d proven that the change wouldn’t break anything. That way we retained a totally stable version. Without this rule, each engineer could have spent days looking for problems that were, in fact, introduced by another engineer.
MSJ: IBM and Microsoft, to put it mildly, have very different corporate cultures and philosophies. How did you mesh the two environments during the development of OS/2?
Letwin: First, whenever we arrived in Boca Raton, we rushed into the bathroom and changed into suits.
But seriously, there were some times when differences of opinions were frustrating. Normally each company builds its own products and calls all the shots. But in a joint development, compromises were necessary. IBM engineers were slowed down by a set of rules which are intended to bring stability to large IBM projects. This naturally caused some conflicts. IBM-ers undoubtedly felt that Microsoft engineers were “wild and crazy guys.”
MSJ: What parts of OS/2 would you do differently if you had the choice?
Letwin: A year ago it became clear we weren’t going to meet our schedules, and we had to go through a simplification procedure. For example, we took out named pipes, a form of inter-process communication to send information to another program (now part of the OS/2 LAN Manager). We’ll put that in another release of the product. Installable file systems were another victim. As a result, the hard disk file system still doesn’t perform as well as we’d all like. These features will be added later. Also, we now know the parts of the system which turned out to be bigger or slower than they should, or which in some other way were a bit disappointing. Naturally we’d give them additional attention. At the end of any project, there are always a few things you’d like to improve. And we will, in the next release.
MsJ: How does it feel to give up your product when it’s ready to go to market?
Letwin: At first, I felt relief when we got the product through testing and it went out the door. The last 20 percent is hard work. You just get tired of the product. The exciting stuff takes place in the earlier stages of the project. The last days are dealing with problem after problem—fixing bugs, making it smaller, making it faster, etc. You’ve worked on it so much there doesn't seem to be much change over the last few months. You’re tired and all you do is deal with problems. In my opinion, it’s the ability to keep working hard during the last part of a long project that separates the great people from the average. In any case, when you finally move on to something else, you feel a mixture of relief that it’s completed, excitement about your new project, and anxiety about how well your efforts will be received.
MSJ: What’s your philosophy of programming?
Letwin: Software is special at our level. Computer programming is 20 percent science, 60 percent engineering, and 20 percent art. The “art” element is the hardest to achieve, and it’s the hardest to explain. When engineers talk about the “elegance” of their work, people usually think that such elegance is non-functional: the engineers are just indulging themselves. This might be the case in mechanical or civil engineering since an elegant bridge carrys no more load than an inelegant one. But it isn’t so in advanced software development because mechanical engineering deals with physical objects and software engineering deals with information. Physical objects have properties independent of human perception, but information doesn’t. In other words, although a bridge is equally functional whether or not it’s elegant, information is much more useful when it’s organized in a clear, flexible and useable form—a form a programmer would call “elegant.” Bridge design is organized to meet just two goals—buildability and final function. Software is designed to meet those goals plus one more. It has to have minimal intellectual complexity and maximum flexibility so that it can be understood by the programmers who will maintain and upgrade it in the future.
GORDON LETWIN: THE CHALLENGE OF THE 286 SPEAKS HIS LANGUAGE
What attracted 34-year-old Gordon Letwin to computers in the late 1960s has also kept him in the forefront of programming today: an insatiable drive to overcome techincal obstacles.
“I’d read about computers in the late 1960s, that they were supposed to be super complicated. Computers held an abstract mystique. And I wanted to understand how that stuff worked,” he says. That was a tough goal at a time when the cheapest computers cost $100,000 and were locked behind glass. Letwin taught himself FORTRAN from a manual, without access to a computer to try it on. While in high school he took some courses at a small private technical college to get free time on the school’s computer.
Letwin studied physics at Purdue, but found he was more interested in getting computer time than in studying physics. Whenever he could, he’d spend his day breaking through the computer’s protective environments instead of studying physics. “Saying that there’s something that I can’t understand, or can’t access because I’m not initiated into some mystery is like waving a flag in front of a bull,” he says. He finally solved this conflict by changing his major to computer science.
A large part of the enjoyment in computing for Letwin is getting immediate results. He likens computers to pinball machines. “You get positive and negative reinforcement right away.” And, being “innately lazy,” he likes the idea of computers doing work for him.
Upon getting his Bachelor’s and Master’s degrees in computer science from Purdue, Letwin went to work for Wintek, a company run by one of his professors. He ended up bidding on a contract to supply Heathkit with a BASIC interpreter, assembler, and editor.
He later moved to Michigan and went to work at Heath but found he didn’t agree with its management’s direction. “I wasn’t happy with the direction of the company, which was run by ‘suits’ who didn’t know computers well but insisted on making the technical decisions anyway.” When his concerns with Heath’s approach at that time weren’t improving, he signed on with Microsoft, then in Albuquerque, New Mexico. “I came here because of Bill Gates,” says Letwin, who met Microsoft’s chairman during sales meetings at Heath.
Letwin became the first new employee when the firm relocated to Seattle about eight years ago. He started working on a BASIC compiler, then a Pascal compiler.
When he became tired of being a “compiler compiler,” he took advantage of his operating system experience to switch to the newly formed operating systems group, starting work on the IBM PC. “We thought the 8086 chip was the first processor powerful enough for general purpose desktop use. We had a vision of an office of the future. We knew that to realize this vision, we’d need a powerful operating system, much more sophisticated then those which were available at that time. Since the only way to insure its existence was to write it ourselves, we got into the operating system business.” The widespread acceptance of MS-DOS makes that statement Microsoft’s platform for its development of the single-user office automation operating system
What computer ads would you like to see in the future? Please comment below. If you enjoyed it, please share it with your friends and relatives. Thank you.