OS Hardening

Summary

In this article the term security is explored and applied to the security requirements of an operating system. A general overview is given on how OS security is defined. Then, the three main sectors on OS security are presented as a high-level overview.

This article also focuses on the following two operating systems:

• Windows operating system
• Linux operating system

OS Hardening

It is to be noted that achieving a completely secure system is almost impossible. What makes this especially challenging is the fact, that a general-purpose operating system has to account for a diverse set of applications running on it. The operating system has only a small influence on changing the behavior of a program, as the source code is mostly controlled by a third party. It is the job of the operating system to manage all of the programs and create a safe environment for the user and the applications.

For instance, typically there are no security measures or checks performed inside the kernel, all kernel components are explicitly trusted, and there are no procedures that separate one element of the kernel from other parts of the kernel. As a result, kernels often do not defend themselves. If the operating system has a security flaw, anyone who is able to exploit it, can take total control of the machine by employing the right software programs. It is challenging to create self-protecting kernels, and performance is frequently severely sacrificed. Nevertheless, both, Linux and Windows operating systems have their advantages and disadvantages and this article gives an overview of available hardening mechanisms both Linux and Windows operating systems, since both play a significant role in daily living.

Benefits of OS Hardening

Making your system more secure can only be a benefit. There is no way this could lead to an disadvantage. Doing OS hardening can enhance your system functionalities, because fewer programs and less functionalities automatically mean that there is less risk of misconfigurations and issues. The second benefit is that the security significantly improves. There is lower risk of unauthorized access, hackings or even malware, if the attack surface is reduced. Also the auditability is simplified, because since there are fewer programs and accounts that means that auditing the environment would also be more transparent.

The goal of OS Hardening

The principal goal of this hardening process is to increase the security standards in the whole world. Analytics found out that in the Linux OS just a little bit above 50% have the newest version of the OS installed. In Windows just 35% which is a pretty small number. For this reason the patch-management will need to be much more involved in the process which will in advance take this low percentages much higher as without it. Also an valid method could be an automatic process which are already included in many OS but also with this process users are not using it properly. The automatic process needs to evolve to the next level which means that the core has to be fully automatic updated when an critical fix went out to public so users who are lazy are also getting the updates that they need.

Security Baselining

Security baselining is the method of putting a minimal set of guidelines in place and configurations for the environment e.g. establishing a minimal Windows device setup [Dun20]. The creation of a baseline offers a minimally defined norm that will assist in assuring a better secure environment if the organisation or the organisational institution deploys systems and devices. Baselines can range from checklists or spreadsheets that someone uses to ensure sure the predetermined security controls have been implemented to a taken snapshot or picture that is already preloaded with the predefined security rules, depending on the size of your business. An aspect of the baseline, it is important to confirm that policies, standards, and processes are in place, are clearly defined, and have received the approval of the leadership and all other stakeholders who are responsible for e.g. data protection. It is essential to have these specified for security, compliance, and auditing reasons. To begin, policies for the firm are to be established and defined. The baselines are then to be constructed using the standards as the framework.

Security Policies, Standards and Procedures

After these baselines are established, policies and procedures is to be developed to apply the baselines and support the achievement of the ultimate result. In the long run, deploying baselines without clear policies, procedures, and a framework is not presumed as effective and leaves the business open to risk. Additionally, having these foundations in place provides a platform with ensuring leadership participation and sign-off, which sends a consistent message to the company about the significance of each associate in its success.[Dun20] It is strongly advised to start with the fundamentals if the business does not already have any rules in place that pertain to, for example, Windows security. In order to protect the devices, at the very least, the following few examples are to be mentioned in a policy [Dun20]:

• Security updates
• Firewall
• Encryption
• A password policy, multi-factor authentification (MFA) and biometrics

A policy that demands that all systems be maintained up to date with the most recent security updates is an example. Policies are followed by standards, which are obligatory and describe the specifics of each policy. In addition to providing details on the technology to be used, standards help maintain consistency within an organization. A few examples of standards regarding windows servers for the proposed elements described in the preceding section include the following [Dun20]:

• Windows Update for Business is to be used to setup all Windows 10 computers, while Windows Servers use either Windows Server Update Services (WSUS) or Azure Update Management. The business use case will specify and store update

schedules.

• On all Windows end-user devices and servers, the Windows firewall is to activate and set up. The connection requirements are to be defined.
• Using BitLocker and/or Azure Disk Encryption, all Windows servers and end-user workstations are to be encrypted.
• PINs and Windows Hello biometrics are to install, and accounts require utilizing passwords with a minimum of 12 characters. Passwords requisite updates yearly and contain capital, lowercase, number, and special characters

The detailed instructions including a step-by-step guidance required to complete a repetitive operation or process are known as procedures. These collections or set of guidances are meant to help with the implementation of the stated policies, standards, and as well as guidelines to achieve a specific aim. Procedures require updates periodically as technology and software versions advance. A third-party tool is also an option with the aim of being more organized and completing procedures. One instance is a program called Nintex Promapp [Pro], which aids in documenting and sharing your company’s procedures [Dun20].

The following four steps are an illustration of a procedure: 1. Set up a new Windows 10 device. 2. Verify that the device is provided with internet connection. 3. Verify the configurations of the device, the device setups, and so on. 4. Verify the device’s compliance. The following figure shows an overview of policies, standards, procedures and baselines and how they are linked to one another.

Operating System Security Evaluation

Creating a completely secure system is hard if not impossible to achieve. Striving to such a goal gets even harder when dealing with enormous complex systems.

There are three main evaluation criteria for testing and classify computer systems:

• TCSEC: Trusted Computer System Evaluation Criteria

developed under the United States Government Department of Defense

• ITSEC: Information Technology Security Evaluation Criteria

published under the Commission of the European Communities

• CC: Common Criteria

These two standards got unified under ISO/IEC 15408 standard.

These standards classify computer systems on a set of criteria ranging from minimal protection (EAL 1) to verified protection (EAL 7).

To bring these requirements into perspective here is a short list, for common OS classification:

• EAL 4+: Red Hat Enterprise 7.1 (2017), Windows XP (2005), Windows 2008 (2008)
• EAL 3: Apple Mac OS X 10.6 (2009)
• EAL 2: Ubuntu 16.04 (2018)

It has to be noted that the examination takes a long time to conduct and only applies to one specific version. This is one of the reasons why such investigations are performed rarely. Additionally, they are very costly, leading to only a marginal gain for the product.

Common Threats

The main security purpose of an operating system is the separation of user data and applications. Other applications or users should not be able to access data they are not authorized to. The operating system has different techniques for achieving this goal. But before we look into the protection and hardening of an operating system, the most prominent attacks are presented. This list of threats is by no means complete and should only give a brief overview.

Application Exploit

Application exploits utilize programming errors to take advantage of the system with the goal in mind to have some kind of privilege escalation and accessing files that should normally not be accessible to a not authorized user.

Buffer Overflow

Buffer overflows are a subtype of an application exploit. As they are one of the more common attacks out there we examine them closer. Buffer overflow is the practice of exploiting unprotected size-unrestrained memory buffers. The goal of such an attack is the modification of the return pointer of a program, enabling the attacker to jump to an arbitrary location in memory. For more info, please visit Buffer Overflows

Race Condition

Race conditions can occur when concurrent processes can read or write on the same memory location. This attack is also known as time-to-check and time-of-use (TOC/TOU) as a shared value is checked and subsequently used to exploit some kind of software bug.

For example, an attacker can create a symbolic link to a file in the precise moment a privileged program wants to write to it. If this symbolic link points to \textit{/etc/passwd}, the privilege program unwillingly overwrites the password file, allowing the attacker root access to the machine

Other Attacks

There are many more techniques attacking the CIA requirements of an operating system. Many malicious actors attack the confidentiality as it usually has the most value to an adversary.

Other attacks on the CIA criteria include SQL injections, web exploits, password cracking, denial-of-service attacks, eavesdropping attacks and malware.

Background of attackers

In the 2019 Verizon data breach investigations report shows that the motives of threat actors were 67% financially motivated, 20% espionage driven, 8% for fun and self-esteem related.

Overall Hardening

OS Hardening is one of the instances in which context the overall system can be secured. That means that OS hardening is just focused on the security measurements in the operating system. To have a overview of the whole tree structure a list below will introduce other instances of system hardening.

• Network Hardening

This type of hardening involves securing the basic communication infrastrucutre of servers and computer systems in a network.

• Physical-Server Hardening

Server hardening involves securing data, ports, components, dunctions and permission of a server.

• Application Hardening

This type of hardening involves updating or implementing additional security measures to protect standard and third-party applications installed on the server. Unlike server hardening, where the focus is on hardening the entire server system, application hardening focuses on the server´s application specifically.

• Database Hardening

Safe Environments

Even though all applications operate in the same memory and on the same CPU, they should not be able to access files that they are not authorized to.

Kernel-Enforced Protection

The kernel does not know the internal functionality of the executing program, therefore it can only modify the layout of the memory and enforce access control rights.

Memory Access Control

By creating non-executable (NOEXEC) memory spaces. This prevents the execution of injected shellcode in the heap or the stack of an application.

Memory Address Randomization

Address Space Layout Randomization (ASLR) introduces randomness of the placement in virtual memory. For an attacker it is therefore hard to know where the location of variables, binary, libraries, heap and stack are in memory, as it is different for each execution.

Compiler-Enforced Protection

This method tackles the problem during the compilation of the program. The compiler leverages the knowledge it has on the structure of the program and could modify it in the way of securing against buffer overflows.

Stack Canaries

The compiler can insert special data, called canaries, into different parts of the program memory that get checked during the program execution. If an attacker tries to overflow a buffer, the possibility of overwriting a canary is very high, since they are placed in strategical positions.

For a more practical visualization please visit Buffer Overflows.

Access Control

Access control on an operating system is needed to preserve the confidentiality of a multi-user and multi-application system. The system has to control who is allowed to access, modify and execute certain data.

Access Control List

In Linux basic file access control is based on the traditional UNIX file model. Each file and directory has a total of 9 bits and some special markers to set the permission. The three bits resemble the privilege to read (r), write (w) and execute (x) a given file.

Dividing the access control into three groups can be quite limiting, especially if there are users that need special permissions and is called simple ACL. Maybe we want to grant permissions to additional users or groups. This is where the traditional system becomes quite constricting.

$ ls -l example.txt 
 -rw-r--r-- 1 root root 0 Jan 31 21:32 example.txt

Therefore, the extension for a extended ACL is provided in Linux per default and enables the definition of additional users (named user) and additional groups (named group). Accessing and viewing the extended ACL in Linux is possible with the command getfacl

// view extended ACL
$ getfacl example.txt

 # file: example.txt
 # owner: root
 # group: root
 user::rw-
 group::r--
 other::r--

// modify or add permissions
$ sudo setfacl -m "u:someuser:rwx" example.txt
$ sudo setfacl -m "g:somegroup:r-x" example.txt 
$ getfacl eammple.txt 

 # file: example.txt
 # owner: root
 # group: root
 user::rw-
 user:someuser:rwx
 group::r--
 group:somegroup:r-x
 mask::rwx
 other::r--

// view with ls command (note the # that indicates the extended acl)
ls -l example.txt 
 -rw-rwxr--+ 1 root root 0 Jan 31 21:32 example.txt

// remove permissions
$ sudo setfacl -x user:someuser example.txt
$ getfacl example.txt 
 # file: example.txt
 # owner: root
 # group: root
 user::rw-
 group::r--
 group:somegroup:r-x
 mask::r-x
 other::r--

// view with ls command
$ ls -l example.txt 
 -rw-r-xr--+ 1 root root 0 Jan 31 21:32 example.txt

Manditory Access Control

The aforementioned method of access control is called the Discretionary Access Control (DAC) method and is the most common mechanism to enforce confidentiality. The idea behind DAC is that the owner specifies who can access an object. This decision is based on the owner's discretion.

The so-called Mandatory Access Control (MAC) tries to tackle the problem with a changed fundamental concept. Instead of letting the user decide the permission of objects, each user is given a certain clearance and each object is given a security classification. Only when a user has a higher clearance level as the required security classification (secret, top-secret, confidential) the system allows access.

Well-known implementations in Linux that implement MAC is SE-Linux and AppArmor.

Reducing the Attack Surface

Reducing the attack surface is a vital part of securing the operating system. As seen before we can use access control to prevent flawed applications to breach confidentiality. We should not rely on this technique entirely and should reduce the possible attacks in the first place.

The most common techniques on reducing the attack surface include:

• Removing unnecessary services
• Secure the network (firewalls)
• Securing the hardware (physical)

Best Practices

The most used best practices which are used in many companies are:

• Analyze the OS version and keep it if possible always up to date
• Prioritize the risks which could happen to the OS from the vulnerabilities
• To go deeper, keep the versions and services clean. This means to always have the newest updates installed, check if the updates are valiable and are also safe from the dev forums
• If a service/program is not needed get rid of it, this will keep the attackers surface at an minimum. More information about the Attacking Surface is in the topic above
• Users and Administrator of a system should have predefined roles which they match. Not used Users should be removed and the guest user is also not a good idea to keep on the OS
• Follow a strategy. This can be done through security templates which have an minimum of security defined which all users must met in order to be able to use this system

General Recommendation

Based on best practices, here are a few general recommendations to keep in mind preventively.

To avoid your system is beeing attacked, some configurations can be preventively done. Independently of the operating system. In the later chapters special tools and configurations for linux and windows hardening will be described in detail. To make sure an IT system is trustworth, it should be ensured that the used hardware and software are coming from a known and trusted source. That includes additional hardware as well like external data storages for example. Also the software should be obtained by trustworthy sources. A software should be downloaded if possible directly from the companies webpage itself. If a software is obtained from online sources then the integrity should be verified. This can be done by checking the hash value which is provided by the manufacturer. To avoid such so called man-in-the-middle attacks software shoudl always transferred over a secure and encrypted channel. TLS for example can be used also to autheticate the issuer.

References

Seminar paper

• File:OS Hardening.pdf
• File:AKITS OS Hardening.pdf

References

• Commission of the European Communities. Information Technology Security Evaluation Criteria (ITSEC): Provisional Harmonised Criteria. Office For Official Publications Of The European Communities, Luxembourg, 1991.
• Luis Franco, Tony Sahama, and Peter Croll. Security Enhanced Linux to Enforce Mandatory Access Control in Health Information Systems. 2008.
• James Graham. Cyber Security Essentials. Auerbach Publications, 2011.
• Andreas Grunbacher. POSIX Access Control Lists on Linux. USENIX Annual Technical Conference, FREENIX Track, 2003.
• Aneesh Kumar, Andreas Grünbacher, and Greg Banks. Implementing an advanced access control model on Linux. 2010.
• Vasudevan Nagendra and Yaohui Chen. Access Control Lists in Linux & Windows. Stony Brook University, 2014.
• P. K. Patra and P. L. Pradhan. Hardening of UNIX Operating System. Int J. of Computer Communication and Technology, 1, 2009.
• Abraham Silberschatz, Peter B Galvin, and Greg Gagne. Operating System Concepts. 2013.
• Peter Silberman and Richard Johnson. A Comparison of Buffer Overflow Prevention Implementations and Weaknesses.
• Verizon. 2019 Data Breach Investigations Report. Technical report, 2019.
• Tevault, Donald A. Mastering Linux Security and Hardening: Secure your Linux server and protect it from intruders, malware attacks, and other external threats. Packt Publishing Ltd, 2018.
• [Dun20] Mark Dunkerley. Mastering Windows Security and Hardening. Packt Publishing, 2020.
• [Pro] Promapp. Business process management - www.nintex.com. Accessed: 2022-11-07

Links

• https://www.redhat.com/en/about/press-releases/red-hat-adds-common-criteria-security-certification-red-hat-enterprise-linux
• https://www.schneier.com/blog/archives/2005/12/microsoft_windo.html
• https://www.niap-ccevs.org/Product/Archived.cfm?par303=Microsoft%20Corporation
• https://www.trentonsystems.com/blog/system-hardening-overview