Operating Systems Security: Threats and Protection Mechanisms

 

An operating system can be defined as “a set of program modules which provides a friendly interface between the user and the computer resources such as processors, memory, Input / Output devices and information” (Madana, 2009). Protecting the operating system is essential in order to control access to computer systems and information stored in them. This paper explores security needs, vulnerabilities and protecting mechanisms in operating systems. The first section deals with an operating System and the need for its security. The second section discusses OS threats and vulnerabilities. The third section covers user-side protection mechanisms and trusted and secured operating systems. Finally, a concluding comparison of the protection mechanisms is provided.

Operating System and the Need for its Security

An operating System is an intermediary between the user of a computer and the computer hardware. It manages the ways applications access the resources in a computer. Major resources include disk drives, CPU, input devices, output devices, main memory, and network interfaces. Without the operating system interaction of users and applications with the hardware are impractical. Operating system assists application programs to be run by users in a very simple and consistent way (Goodrich and Tamassia, 2011). It also allows multiple users with different level of access to the same computer resources.  Operating systems permits not only a single application but multiple applications to run at the same time (multipletasking). The kernel, being the core component of the operating system, handles the management of low-level hardware resources, which include CPU, memory and input/output devices.

The complex tasks of operating system, such as multipletasking as well as the allowance of multiple users to computer resources, calls for security needs.  Computational resources should be protected from malicious or accidental application damages caused by external threats as well as vulnerabilities in the operating systems. Each running application must also be protected from being interfered by other malicious applications. At the same time, since each potential user has unique needs and rights with respect to computer resources, the operating system has to make sure those needs and right are well respected.  

The need for protection arises not just from sharing of the processor among users, but also from the sharing of memory, I/O devices (disks and printers), program applications, and data (Stalling, 2011). The operating system is required to balance the need to allow sharing and with the need to protect the resources of individual users. While sharing boosts the utility of the computer system, protection and security deal with the regulations of user access, information flow and certification. The following sections discuss major threats and vulnerabilities of operating system and major mechanisms of providing operating system (OS) security (traditional and trusted).  

Operating System Security: Threats and Vulnerabilities

The increasing trend in the use of resource sharing systems and computer networks has increased threats to information and information system. Among the general threats to be addressed in the area of security are organized and intentional attempt to obtain economic or market information from competitive private sector or government, unintentional or accidental acquisition of market, economic or individual information, fraud through illegitimate access to data banks, as well as invasion and intrusion on individual rights by the government and intelligence community (Branted, 1978).

Most attacks are often after information or data, which is found in a computer with operating system. Thus, it’s the operating system which is going to be exploited in order to obtain the information from an organization. In order to have a clear understanding of threats in operating system security, requirements for its security should be well stated. Those requirements include confidentiality, integrity, availability and authenticity. While confidentiality requires system information only to be read- accessible to authorized entities, integrity necessitates the computer system assets to be modified only by authorized parties. Availability seeks system assets to be available to authorized users and authenticity requires a computer system to verify user’s identity (Stallings, 2011).

While security concerns of computer system assets encompass hardware, software and data, the major problem is data security. Data availability, confidentiality and integrity are the major security. Accidental or malicious destruction of data file, unauthorized reading of data files or database, and modification of data files are the major concerns behind the operating system security. Operating system security include monitoring and protecting the process that is running the computer, protecting its memory and file system, and protection of application programs operating at the application layer. Some of the major attacks are discussed below.

A typical example of attack on the process running the computer is the hibernation attack, which happens when a machine goes into a power-off state (hibernation). When a user closes a laptop computer and puts it into hibernation, an attacker can easily copies the hibefil.sys file and discovers any unencrypted password that was stored in memory during hibernation (Goodrich and Tamassia, 2011). An attack on virtual memory, a tool which allows multiple processes in the memory by creating swap files, is another security concern. An attacker can boot the machine to another operating system via external media and can view the swap file and expose sensitive information. A dictionary attack can be used to guess encrypted passwords kept in the operating system. An attack on an application programs, known as a buffer overflow, allows an attacker to obtain control of the entire process on the machine. A Trojan horse attack is a typical example of malicious program which calls for a secured operating system.  It’s a malware program which appears to perform some useful tasks, but which also has a malicious effect (Stallings, 2011).

Most of the attacks arise from the inherent weaknesses or vulnerabilities of the operating systems. Security comparison of the two major operating systems, Windows and Linux, indicate that there are some security issues worth discussing in relation to their authentication, authorization, and auditing capabilities. Password protection is currently the primary source of authentication in both. However, the Linux password encryption scheme is more effective than Windows, as Linux uses a password salt, a random value generated and added to the user's password before encryption. Therefore, brute force attack is more difficult in Linux (Nemati, 2008). Malicious software running in user-space is the most common cause of security exploits in both. Particularly, the buffer overflow attack has been used extensively on Windows platforms. The security design principle, complete mediation is thorough and complete in both operating systems. However, compliance with the principle of least privilege is a big concern with both operating systems. Logging which is the foundation of good auditing. Both Linux and Windows have good logging capabilities. However, network based logging capabilities are not currently part of the basic Windows and Linus operating system, despite the increasing attack via the network interface. The next section deals with some of the mechanisms by which operating systems are protected from attacks and how trusted systems are used.
Mitigation or Protection Mechanisms

In order to design security measure for the various threats to computer system and operating systems in particular, a number of design principles are identified. These include least privilege, economy of mechanism, acceptability, complete mediation and open design (Saltzer & Schroeder, 1975). The Least Privilege principle states that each program and user of an IT system should operate with minimum set of privileges necessary to function properly. In the Economy of mechanism, simplicity of design and implementation of security measures are sought. While Acceptability stresses easy to use interface, Complete mediation principle emphasizes on the idea that every access to the resource must be checked against access control mechanism. Open design principle focus on keeping the security design and architecture to be made public. The first part of this section addresses operating system protection mechanisms on the user side. The second part concentrates on trusted operating systems designs and their implementation.

User-side Protection Mechanisms

Protection of Memory

 

For the sake of correct functioning of various processes as well as security, protecting the main memory will be essential. This is especially true in a multiprogramming environment (Stalling, 2011).  A virtual memory scheme helps the separation of memory. Using segmentation or paging, or the two in combination, main memory is managed effectively, and implementation of protection and sharing policies made easy. Virtual memory allows more space or larger than the RAM memory for multiple processes to run effectively. At the same time, viewing the contents of virtual memory files is prevented while the operating system is running. In addition, the risk of exposing file content can be mitigated using with hard disk encryption (Goodrich and Tamassia, 2011).

User-Oriented Access Control

 

Measure taken to control access in data processing is achieved in either user-oriented or data-oriented means.  User-Oriented Access Control is sometimes called authentication. The most common technique used here is the user logon.  User logon requires the knowledge of both user ID and password. A shared system or server allows a user only if the system knows the user identifier, ID, and the user knows the password linked to that ID. This system is infamously undependable. Passwords can be forgotten or accidentally revealed. Hacker can skillfully guess the ID and brute force the password.

We may have either a centralized or decentralized user access control. In centralized environment, who is allowed to use the network and what to be connected is determined by the network. In decentralized approach, the destination host carries out the logon procedure. However, to protect host-specific resources, two levels of access control are used in many networks. In the two-level way, while the network provides protection by restricting access to authorized users, individual hosts provide the logon procedure.

Data-Oriented Access Control

 

Once logon is successful, granting the user to one or a list of hosts or applications is not satisfactory for system protection where we have of plenty of sensitive data and applications. In data-oriented access control, following a successful logon, and after operating system grants permission to a user to access a file or an application, a dbase management system makes a decision on each individual access attempt.  The grant decision is dependent not only on the user identity, but on the specific portion of data to be accessed. A general model, an access matrix, is used to control access by files or data base management system. In the model we have three basic elements: subject, object and access rights. A subject is an entry capable of accessing an object. An object is anything to which access is controlled. And access entry is the way in which an object is accessed by a subject, often read, write and execute. In access matrix table each row represents a subject, each column represents an object and each entry in the matrix indicates the access rights of a specific subject for specific object (Pfleeger and Pfleeger, 2009).

Operating System Mode (OS Rings)

 

One protection technique used in all operating systems is based on the ‘mode’ of processor execution. We have two distinct operating system modes, the kernel mode and the user mode.  In most operating systems, applications are separated from the operating system itself. The kernel mode or system mode runs in a privileged processor mode, with access to the system data and hardware. The user mode or application code runs in a non-privileged, limited access to system data and no direct access to hardware (Russinovich et al, 2009). When a user makes a call for a system service, the mode is set to the kernel mode. When the system service completes, the operating system switches back to user mode and allows the user to continue.

In the kernel mode, the operating system has a complete control of the processor and all its instruction, registers and memory (Stallings, 2011). The operating system can be explained in terms of rings to clearly demonstrates how the protection works. Ring 0 is a system memory where kernel and operating system resides and Ring 2 where user application exists (and Ring 1 belongs to device drivers). If Ring 0 is compromised by an attacker, all access to system resources can be controlled by their malicious software (University of Maryland University College, 2011). For these reasons, full level of control is not required and should not be allowed for user programs. So far, operating system protection from the user’s point of view has been discussed. The next section discusses trusted operating systems designs, their functions and limitations.

Trusted Systems Protection

An operating system is considered to be trusted if there is confidence that it provides the four services (memory protection, file protection, general object access control and user authentication) consistently and effectively (Pfleeger and Pfleeger, 2009). Trusted system protection coverage extends from initial boot process and kernel, application and file system protection, full disk encrypting to a combination of software and hardware trust solutions. This section discusses three major modules: Multiple Independent Levels of Security (MILS), Trusted Platform Module (TPM) and Trusted Computing Base (TCB).

Multiple Independent Levels of Security (MILS)

What makes the MILS distinct is that it’s an operating system which is built from the very beginning with security in mind.  MLIS is considered to be high assurance architecture for handling information of different classification level. It’s designed to protect against malicious software, internal errors, and system failures. There are partitions which run in a separate environment, hampering interactions between system cases. This is accomplished using the following key security policies, namely, information flow, data isolation, periods processing and damage limitation (www.ois.com/products, 2011). Policy implementation is the responsibility of the middleware layer.

The flow of Information is limited between partitions. And if it’s required it can be processed only after explicit request is made through the middleware layer. Data isolation makes sure that private data remain private. MLIS’s period processing ensures the microprocessor of a system is cleaned (using encryption) before switching from one application to another. Data limitation mitigates a failure in one partition, as breakdown in one partition will not cascade into another one.

In general, the MLIS‘s architecture benefits include reduction in hardware component, flexibility in  information control and management, relatively cheaper of highly secured systems development, and less need for redesigning systems to meet security standards (University of Maryland University College, 2011). The concept of separation is the biggest advantage of MILS.

Trusted Platform Module (TPM)

The Trusted Computing Group (TCG), a non-profit group, has been working to improve trust and security in today's open computing platforms.  The group defined trust as "the expectation that a device will behave in a particular manner for a specific purpose"(Krutz, & Fry, (2009). Believing that software alone is not strong enough to protect information system; they utilized both hardware and software to develop the Trusted Platform Module (TPM). TPM is a hardware-based trusted protection mechanism designed to protect the security and privacy in a computer system.

TPMs are usually installed on PC motherboards and are designed to protect cryptographic keys and authentication processes and provide certification. The TPM is an element that can securely generate, store, and manage cryptographic keys which can be used as a private key to eventually decrypt the data. Encrypted data cannot be decrypted unless the key is provided by the secure TPM following appropriate authentication. A secret and unique RSA key is built into the TPM chip during its production. The key can be used to verify the authenticity of other systems with TPM chips.

Authentication boot service is done by validation of codes through the use of digital signatures and hash values during the booting stages (reading ROM, referring to the master boot block, locating the operating system). In every stage, the Trusted Platform Module checks integrity. And whenever an application is loaded or hardware needs to be configured, approved listing is used to check the system. And it makes sure, if applicable, digital signature is done before configuring or loading is performed. Thanks to its inbuilt prevention mechanism, TPM is not susceptible to ‘dictionary-based’ attacks.  However, a ‘cold boot attack’ showed that encryption keys remaining in memory can be recoverable (University of Maryland University College, 2011).

Trusted Computing Base (TCB)

TCB is a name given to all of the systems in the network in the trusted operating system necessary to enforce the security policy. It consists of all the parts of the trusted system on which we depend to enforce the policy. The security of the whole system thus depends on TCB. And it is indispensible for the TCB to be thorough and correct to fulfill the security policy.  The TCB does not address only the operating system. It covers hardware, software and firmware. The protected hardware includes processor, memory, register, and I/O devices. The software and firmware protected include operated system (hardened kernel), configuration files, shell/windowing system and peripheral devices firmware (Pfleeger and Pfleeger, 2009; University of Maryland University College, 2011).  

The main functions of the TCB are process activation, memory protection, execution domain switching and I/O operation (Harris, S., 2010). A process is "activated" when its request is made, enabling it to interact with the CPU. A process is "deactivated" after execution by the CPU or when CPU is called by other priority. Memory protection is done using TCB monitoring of each domains code and data to ensure secrecy and integrity. Execution domain switching refers to when the CPU go from execution of instruction in user mode (less trusted) to privileged mode or back. The TCB makes sure this happen properly or less trusted process will be executed in privileged mode and system resources will be compromised.

 Conclusions

From the user’s perspective, for an operating system to be “secure” the operating system should provide the following services: memory protection, file protection, general object access control and user authentication. And an effective and consistent provision of such services make an operating system “trusted”.  The MILS architecture having inherent partition protection means less hardware requirement, saving space and power. It is also cheaper and faster for development, with easier management and control. MILS makes it easier for commercial of the shelf (COTS) components to be integrated with less re-architecting. TPM is a hardware-based trusted protection. It makes sure that the system secured through three services, namely, certification, encryption and authenticated boot. Its inbuilt protection mechanism makes it invulnerable to dictionary-based attacks. TCB is the trusted operating system necessary to enforce the security policy which comprises hardware, software and firmware. Evaluating the security features of a ‘secure’ or ‘trusted’ operating system requires reviewing requirements, designs, implementation, as well as evidence of assurance for each trusted computer system.

 

References:

Anderson, R. (2008). Security engineering – A guide to building dependable distributed systems (2nd

                ed.). New York, NY: John Wiley & Sons Publishing, Inc. Chapter 18, “API Attacks”

Beuchelt, G. (2009). Chapter 5, Unix and Linux Security In Vacca, J. R. (Ed.), Computer and information security handbook. Boston, MA: Morgan Kaufmann Publishers.

Goodrich, M. and Tamassia,R. (2011). Introduction to Computer Security. Chapter 3, Operating Systems Security pp. 114-165.

Krutz, R. L. & Fry, A.J., (2009). The CSSLP prep guide: mastering the certified secure software

lifecycle professional. Retrieved from

http://common.books24x7.com.ezproxy.umuc.edu/toc.aspx?bookid=32022.

Harris, S., (2010). Cissp all-in-one exam guide, fifth edition. Retrieved from

http://common.books24x7.com.ezproxy.umuc.edu/toc.aspx?bookid=35956.

Madana K., (2009). Operating systems made easy. Retrieved from

                http://common.books24x7.com.ezproxy.umuc.edu/toc.aspx?bookid=34048.

Nemati, H., (2008). Information security and ethics: concepts, methodologies, tools, and applications. Retrieved from http://common.books24x7.com.ezproxy.umuc.edu/toc.aspx?bookid=22649.

Pfleeger, C. and Pfleeger, S.L., (2007). Security in Computing, 4th ed., Prentice Hall, Englewood Cliffs, NJ.

Russinovich, M. E., Solomon, D. A. & Ionescu, A., (2009). Windows Internals, fifth edition.

Retrieved from http://common.books24x7.com.ezproxy.umuc.edu/toc.aspx?bookid=24534.

Salzer, J. and Schroeder, M., (1975). The protection of Information in Computer System. Proceedings of the IEEE. 63, 9, 1278-1308.  

Santana, M. (2009). Chapter 6, Eliminating the Security weakness of Linux and Unix Operating Systems. In Vacca, J. R. (Ed.), Computer and information security handbook. Boston, MA: Morgan      Kaufmann Publishers.

Stallings, W., (2011). Operating System Security. In H. Bidgoli (Ed.), Handbook of information security, volume 2. Part 3: Foundations of Information, Computer and Network Security, New York, NY: John Wiley & Sons, Inc.