Chapter 2: Core Operating System Services and Architecture

Introduction

An operating system (OS) exists to provide a bridge between users and the raw hardware of a computer. The way an operating system is organized internally varies tremendously across different systems, but all modern operating systems must accomplish a common set of goals. In this chapter, we explore the operating system from three distinct viewpoints: first, as a collection of services available to users and programs; second, as an interface through which users interact with the machine; and third, as a structured set of components that work together. Understanding these perspectives gives us insight into both how operating systems are designed and why they work the way they do.

2.1 Operating-System Services

An operating system provides a collection of services that fall into two broad categories: those that directly benefit users and those that improve the efficiency and reliability of the system itself.

User-Facing Services

The first group of services exists to make the computing environment more usable and practical for people and their applications.

User Interface. Nearly every operating system provides some way for users to interact with it. The most common form today is a graphical user interface (GUI), where users manipulate visual objects called icons using a mouse or trackpad to select files, launch programs, and open menus. Mobile devices use a touch-screen interface, allowing users to swipe fingers across the screen or tap buttons to make selections. Older systems and specialized environments often employ a command-line interface (CLI), where users type text commands to instruct the system what to do. Many modern systems provide all three interfaces, letting users choose their preferred interaction style.

Program Execution. The operating system must be capable of loading a program from storage into memory and allowing it to run. When the program finishes—either normally or due to an error—the OS must handle cleanup and return control to the system.

Input/Output Operations. Programs rarely work in isolation; they need to read from and write to devices like disks, keyboards, and network interfaces. Since allowing programs direct control over devices would create chaos and security risks, the operating system acts as a gatekeeper, managing all I/O requests and ensuring safe, orderly access to devices.

File-System Manipulation. Programs need the ability to create, delete, read, and write files. They must also be able to search for files by name and manage permissions that control who can access which files. Different operating systems offer different file systems, each with different characteristics and performance trade-offs.

Communications. When one process needs to exchange information with another—whether on the same machine or across a network—the operating system provides the communication infrastructure. This may be implemented through shared memory, where multiple processes read and write to a common region of memory, or through message passing, where the OS relays information between processes in carefully controlled packets.

Error Detection and Handling. Hardware failures, software bugs, and unforeseen conditions happen constantly. The operating system monitors for errors in the CPU, memory, I/O devices, and user programs. When an error occurs, the OS decides whether to attempt recovery, terminate the offending process, log the problem for later diagnosis, or in serious cases, halt the entire system.

Error Detection: The operating system's responsibility to identify hardware failures, memory faults, device problems, and software errors, and to take appropriate corrective action to maintain system stability.

System-Efficiency Services

The second group of services exists to maximize how efficiently the computing resources are used when multiple processes are competing for access.

Resource Allocation. When multiple programs run simultaneously, the operating system must decide how to divide the processor, memory, and I/O devices among them. Different resources require different allocation strategies. The CPU uses scheduling algorithms that consider processor speed, the priority of waiting processes, the number of processing cores, and many other factors. Peripheral devices like printers and USB drives use more straightforward allocation queues.

Accounting and Logging. The OS tracks which programs use how much of each resource. This information serves multiple purposes: it can be used to bill users for their computing time, to identify bottlenecks that indicate where system upgrades are needed, or to create historical records for debugging.

Protection and Security. In any system with multiple users or multiple running processes, one user's or process's activities should never interfere with another's. Protection means controlling which users and processes can access which resources. Security extends this further, protecting against external threats through authentication (requiring users to prove their identity, typically with a password) and by preventing unauthorized access to network interfaces and other entry points. A secure system requires defenses at every layer—a chain is only as strong as its weakest link.

Protection: A system property ensuring that access to all system resources is controlled and that one process cannot interfere with another process or with the operating system itself.

Security: The set of mechanisms an operating system uses to authenticate users, control access to resources, defend against external attacks, and maintain the integrity of the system.

📝 Section Recap: Operating systems provide two categories of services. User-facing services include the interface for interacting with the system, the ability to run programs, access to files and I/O devices, inter-process communication, and error handling. System-efficiency services include resource allocation among competing processes, accounting and logging for management purposes, and protection and security mechanisms to isolate processes and prevent unauthorized access.

2.2 User and Operating-System Interface

Different users prefer different ways of interacting with a computer, and different tasks are sometimes better suited to different interfaces. Modern operating systems typically support multiple interface styles simultaneously.

Command-Line Interpreters

A command interpreter (also called a shell) is a special program that runs when a user logs in or opens a terminal. Its job is simple but crucial: read a command typed or provided by the user, figure out what program should execute that command, and run it. On UNIX and Linux systems, users can choose among multiple shell implementations, such as the Bourne-Again shell (bash), the C shell, and the Korn shell. These shells provide similar core functionality, and users typically pick based on personal preference.

There are two fundamentally different architectural approaches to implementing commands. In the first approach, the command interpreter itself contains the code to execute each command. This means every new command requires adding new code to the interpreter, which makes the interpreter grow in size. In the second approach—used by UNIX and most modern systems—most commands are implemented as separate programs. The shell simply looks up the command name, loads the corresponding program from disk into memory, and executes it with the parameters the user provided. This design is much more flexible, because programmers can add new commands simply by creating new program files without touching the shell at all.

Graphical User Interfaces

Rather than typing commands, users can interact with a GUI by moving a mouse pointer over visual elements (icons, windows, buttons) and clicking. This approach emerged from research in the early 1970s at Xerox's PARC facility and became widespread with the Apple Macintosh in the 1980s. Microsoft Windows brought GUI functionality to PCs, and modern operating systems like macOS and various Linux desktop environments all offer rich graphical interfaces.

The advantage of a GUI is that it is generally more intuitive for casual users. The disadvantage is that graphical interaction is often slower for power users and system administrators who need to perform complex operations. Graphical interfaces also make it harder to automate repetitive tasks, whereas command-line shells support scripts—text files containing sequences of commands that can be executed like programs.

Touch-Screen Interfaces

Mobile devices like smartphones and tablets typically use a touch-screen interface, where users interact by gestures: tapping, swiping, pinching, and dragging their fingers across the display. These devices usually simulate a keyboard on screen rather than including a physical keyboard.

Interface Choice and Trade-Offs

The choice of interface is ultimately a matter of personal preference and task fit. Power users and system administrators often prefer command-line interfaces because they are faster for experienced users and more amenable to automation through scripts. Casual users and most consumers prefer graphical interfaces because they are more discoverable and require less specialized knowledge. Most modern operating systems (including macOS with its Unix kernel and recent versions of Windows) support multiple interface styles, allowing users and administrators to work in whatever style suits them.

📝 Section Recap: Operating systems provide multiple interface styles: command-line interpreters (shells) for text-based interaction, graphical user interfaces for visual interaction, and touch-screen interfaces for mobile devices. Command-line shells typically delegate command execution to separate programs, making them easy to extend. Different interfaces suit different users and tasks; power users often prefer command-line efficiency, while casual users often prefer graphical discoverability.

2.3 System Calls

While the user interface is what users see, the actual mechanism by which programs request services from the operating system is the system call. A system call is a formal request from a program running in user mode asking the operating system (running in privileged kernel mode) to perform some action on its behalf.

Why System Calls Matter

Even simple programs make heavy use of system calls. A program that just reads from one file and writes to another must make multiple system calls: to accept file names from the user, to open and create files, to read data, to write data, to close files, and to report results. More complex programs may make thousands of system calls per second.

Application Programming Interfaces

Most programmers do not work directly with system calls. Instead, they use an Application Programming Interface (API), which is a set of functions provided by the operating system (through a library) that programmers can call from their code. The API specifies what each function does, what parameters it expects, and what it will return.

Three widely used APIs are the Windows API for Windows systems, the POSIX API for UNIX-like systems (including Linux and macOS), and the Java API for programs running on the Java Virtual Machine. Behind the scenes, functions in these APIs invoke the actual system calls in the operating system kernel. For example, the Windows API function CreateProcess() actually invokes the NTCreateProcess() system call in the Windows kernel.

Using an API has several advantages over calling system calls directly. First, it promotes portability: a program written using a standard API can potentially run on any system that supports that API, without recompilation. Second, APIs are typically easier to use than raw system calls because they provide higher-level abstractions. Finally, the run-time environment (RTE)—the complete software needed to run applications in a given language—handles the translation from API function calls to kernel system calls, isolating the programmer from system-call details.

System Call: A formal request issued by a user-mode program asking the operating system kernel to perform a privileged operation on its behalf.

Application Programming Interface (API): A set of functions and conventions that allow application programmers to request operating system services without needing to know the details of how those services are implemented.

Passing Parameters to System Calls

When making a system call, the program must communicate which system call it wants and what information it needs. Parameters can be passed to the system call using one of three methods:

1. Register passing: The parameters are placed in CPU registers, and the kernel reads them from the registers. This is fast but limits the number of parameters (there are a limited number of registers).

2. Block or table passing: When there are more parameters than available registers, the program stores the parameters in a block of memory and passes the address of that block in a register. The kernel reads the block to access all the parameters.

3. Stack passing: Parameters are pushed onto the system stack, and the kernel pops them off as needed.

Different operating systems prefer different methods. Linux uses a hybrid approach: if there are five or fewer parameters, it uses registers; if there are more than five, it uses the block method. The advantage of block or stack methods is that they can handle an arbitrary number of parameters.

📝 Section Recap: System calls are the mechanism by which programs request services from the operating system kernel. Most programmers use Application Programming Interfaces (APIs) rather than calling system calls directly, gaining portability and ease of use. The run-time environment translates API function calls into system calls. Parameters are passed to system calls using registers, memory blocks, or the stack, depending on how many parameters need to be passed.

2.4 Types of System Calls

System calls can be grouped into six major categories based on the service they provide.

Process Control. Programs must be able to terminate themselves normally or abnormally. If a program terminates abnormally due to an error, the operating system may generate a memory dump (a record of the program's memory contents) for later diagnosis by a debugger—a tool that helps programmers find and fix errors. The operating system must also manage process states, allowing programs to halt temporarily and resume later, and allowing one process to wait for another to finish.

File Management. Programs need to create, delete, open, and close files. They must be able to read and write file contents, seek to a particular position in a file, and obtain information about files (like their size or creation date).

Device Management. Just as with files, programs need to open and close I/O devices, read from and write to them, and request special I/O device operations. Some devices can be treated almost like files (you can read from a disk), while others are more specialized.

Information Maintenance. Programs may ask the operating system for system information: the current time and date, system statistics, process information, or configuration parameters. The OS also provides calls to set this information (for example, setting the system clock or adjusting resource limits).

Communications. When processes need to exchange information, the operating system provides system calls for creating communication channels (pipes or network sockets), sending and receiving messages, and managing the communication connection.

Protection and Security. The operating system provides calls to control access permissions for files and devices, and to manage the user authentication system.

📝 Section Recap: System calls fall into six categories: process control (starting and stopping programs), file management (creating, reading, and writing files), device management (controlling hardware devices), information maintenance (managing system information), communications (allowing processes to exchange data), and protection/security (controlling access to resources).

2.5 System Programs and Services

Beyond system calls, the operating system provides a collection of system programs that offer useful utilities for users and programmers. System programs are applications that come with the operating system and typically provide functionality that users expect to find available.

File manipulation programs allow users to create, delete, list, print, and copy files and directories. Programmers use compilers to translate source code into executable form, and assemblers to convert assembly language into machine code. Modern operating systems also provide interpreters for interpreted languages like Python and Ruby, which read and execute source code directly without a separate compilation step.

System status programs allow users to check system statistics, view running processes, examine network connections, and inspect system configuration.

Programming support includes not just compilers but also debuggers (programs that help find bugs) and system libraries that provide commonly needed functions.

The operating system itself must be loaded when the computer starts up. The process of starting an operating system is called booting, and it involves special firmware code that initializes hardware and then loads the OS kernel from storage into memory.

System Program: An application provided with the operating system that offers useful functionality to users and programmers, such as file editors, compilers, and diagnostic tools.

2.6 Why Applications Are Operating-System Specific

A program compiled for one operating system typically cannot run on a different operating system, even if both systems run on the same processor hardware. The reason is that each operating system provides a unique set of system calls. But even if system calls were standardized, other barriers would prevent portability.

There are three main approaches to making programs available across multiple operating systems:

1. Interpreted Languages: A program written in an interpreted language like Python or Ruby can run on any system with an interpreter for that language. The interpreter reads the source code and translates each instruction to the native system calls. This approach suffers from performance penalties compared to native compilation, and interpreters typically support only a subset of each operating system's features.

2. Virtual Machines: A program written in a language like Java runs inside a virtual machine, which is a software simulation of a complete computer. The Java Virtual Machine is part of Java's runtime environment and runs on many operating systems. The same compiled Java bytecode can run on any system with the appropriate virtual machine, providing true portability. The trade-off is that virtual machine execution is slower than native compilation.

3. Binary Compatibility: Some operating systems are designed to be binary compatible, meaning they run the same executable files as another OS. For example, many Linux systems can run Windows applications through compatibility layers, or some BSD systems are compatible with UNIX binaries.

📝 Section Recap: Applications compiled for one operating system cannot usually run on another because each operating system provides different system calls and may format executable files differently. Portability across operating systems can be achieved through interpreted languages, virtual machines, or binary compatibility layers, each with different trade-offs between portability and performance.

2.7 Operating-System Design and Implementation

When designing a new operating system, the first step is to establish clear goals. Is the system intended for a supercomputer or a smartphone? Should it prioritize throughput (how much work gets done) or responsiveness (how fast individual operations complete)? Should security be the highest priority, or should ease of use? Different goals lead to different design choices.

Once goals are set, designers typically separate mechanism (how something is done) from policy (what decision is made). Mechanisms are relatively permanent—they change slowly as hardware and fundamental principles evolve. Policies, on the other hand, can change based on usage patterns and requirements. For example, a memory management mechanism might implement virtual memory and paging, but the policy determines how much memory each process gets. Separating mechanism from policy makes operating systems more flexible and easier to adapt.

Designers also consider whether to build a new system from scratch or to leverage and learn from existing systems. Many modern operating systems are built by modifying and extending established designs rather than starting completely fresh.

Mechanism: The techniques and structures used to implement operating system functions—relatively permanent design elements.

Policy: The rules and decisions about how mechanisms are used—often changeable based on system requirements and usage patterns.

2.8 Operating-System Structure

The internal organization of an operating system can follow several different architectural approaches, each with different trade-offs between simplicity, performance, and maintainability.

Monolithic Kernels

In a monolithic kernel design, all operating system functionality is compiled into a single, large kernel program. The kernel contains all code for process management, memory management, I/O, file systems, and everything else. When procedures call each other, they use normal procedure calls; there is no protection between different parts of the kernel.

Monolithic kernels are simple and efficient because there is minimal overhead from crossing protection boundaries. However, they are difficult to maintain because they are so large and complex, and a bug in any part of the kernel can crash the entire system.

Layered Approach

A layered operating system organizes the kernel into layers, where each layer provides services to the layer above it and uses services from the layer below it. At the bottom is the hardware; above it are layer-by-layer abstractions for increasingly complex services. Each layer uses only the layers below it, which creates a clean separation of concerns.

The advantage of layering is that it is easier to design and debug because each layer has a well-defined interface and only needs to know about the layers below it. The disadvantage is that it can be inefficient: even simple operations may need to pass through many layers. Also, strict layering can be difficult to achieve in practice because higher layers sometimes need to bypass lower layers for performance.

Microkernel Design

In a microkernel design, only the most essential services are implemented in the kernel: typically process management, memory management, and inter-process communication. Most other services (file systems, device drivers, networking) are implemented as separate user-level programs called servers that communicate with each other and with applications through message passing using the microkernel's IPC mechanism.

The advantage of the microkernel approach is that the core kernel is much smaller, easier to maintain, and easier to extend. A bug in a device driver or file system server won't crash the kernel—only that service fails. The disadvantage is that the system is more complex, and message-passing communication is typically slower than direct kernel calls.

Monolithic Kernel: An operating system design in which all functionality is included in a single kernel program, providing efficiency but making maintenance difficult.

Microkernel: A minimalist operating system design where only essential services run in the kernel, and other services run as user-level programs that communicate through inter-process communication.

Modular and Hybrid Approaches

Modern systems often use loadable kernel modules, which are program files that can be linked into the kernel while the system is running. This approach combines some of the simplicity of monolithic kernels with some of the flexibility of microkernels. Critical services are in the kernel for efficiency, but less critical functionality can be loaded and unloaded as needed.

Many contemporary operating systems use a hybrid approach, incorporating elements of multiple architectural styles. The Linux kernel, for example, is primarily monolithic but supports loadable modules. Windows uses a hybrid approach combining some monolithic design with loadable drivers and user-level services.

📝 Section Recap: Operating systems can be organized using different architectural approaches. Monolithic kernels are simple and efficient but difficult to maintain. Layered designs provide good separation of concerns but can be inefficient. Microkernel designs make the kernel small and modular but introduce communication overhead. Modern systems often use hybrid approaches combining elements of multiple strategies or support loadable modules for flexibility.

2.9 Building and Booting an Operating System

Creating an operating system requires careful planning and coordination of many parts. Developers must decide how to configure and customize the system for its intended use. Some operating systems are stored in firmware (permanent, unchangeable storage), while others are stored on disk and loaded into memory when the computer starts.

The boot process (or bootstrapping) begins when the computer is powered on. First, firmware code in read-only memory (ROM) initializes the hardware: it checks what devices are present, tests memory, and identifies the disk where the operating system is stored. This firmware code (the bootloader) then searches for the operating system kernel on the specified disk, loads it into memory at a specific address, and transfers control to it. The kernel then initializes its own internal data structures, enables interrupts, and starts the first user-level processes.

Different systems handle this differently. On personal computers, the bootloader is typically simple and just loads the kernel. On embedded systems or systems with special requirements, the bootloader might handle more complex initialization.

Booting: The process of initializing a computer and loading the operating system kernel into memory so the system can begin normal operation.

2.10 Operating-System Debugging

Finding and fixing errors in an operating system is particularly challenging because an error in the OS kernel can cause the entire system to become unstable or unresponsive. Operating system developers use specialized tools and techniques for debugging.

Failure analysis begins with observing that a system has behaved unexpectedly. If a program terminates abnormally, the operating system may capture a core dump (a snapshot of the program's memory at the time of failure). Developers analyze core dumps using a debugger, a specialized tool that allows inspection of memory, execution of programs one instruction at a time, and observation of program state. Debuggers can examine what code was executing, what values were in variables, and what system calls were being made.

For more complex OS-level issues, developers use performance profilers and other monitoring tools that track resource usage, system call patterns, and other metrics. Some operating systems include built-in logging and tracing facilities that record what the system is doing so developers can later examine the trace and understand what went wrong.

Modern systems often use crash dumps from production systems to understand real-world failures and improve reliability.

Debugger: A software tool that allows programmers and developers to observe the execution of a program step-by-step, inspect memory and variables, and identify the cause of errors.

📝 Section Recap: Debugging operating systems requires specialized tools because kernel errors can crash the entire system. Developers use core dumps, debuggers, performance profilers, and logging facilities to analyze system behavior and identify bugs. Production crash dumps provide valuable information about real-world failures.

Summary and Key Takeaways

An operating system is fundamentally a collection of services—some that users directly interact with, and others that keep the system running efficiently and safely. These services are accessed through system calls, which form the boundary between user-mode programs and the kernel.

The operating system's interface with users has evolved from command lines to graphical and touch-based interactions, and modern systems typically support multiple interface styles simultaneously.

The internal organization of the operating system significantly affects its properties. Monolithic kernels prioritize efficiency but sacrifice modularity. Microkernel designs prioritize modularity and reliability but introduce communication overhead. Hybrid designs attempt to balance these trade-offs. Most modern operating systems are not purely one architectural style but instead combine elements strategically.

The journey from pressing the power button to a fully functional operating system involves careful coordination between firmware, bootloaders, and the kernel itself. And when things go wrong—which they inevitably do—specialized debugging tools and techniques help developers understand failures and improve system reliability.

Understanding these concepts provides the foundation for grasping how modern computers work and sets the stage for exploring more detailed topics like process management, memory management, and I/O operations in subsequent chapters.