OPERATING SYSTEM

SECURITY
Information Security BMIS 403

By Prasad Peiris
 What we studied last time?
Program Flaws

Non-malicious Flaws Malicious Flaws Counter Measures

Virus Prevention/Detect
Buffer Overflows
ion/Reaction

Incomplete Mediation Worm Antivirus Software

TOCTTOU Errors Backdoor Advance

Techniques

Logic Bomb

Trojan Horses
What we will study today?
OS Security

OS Sec Goals
OS Overview OS Sec Environment Top OS Vulnerabilities

Goal 1 Goal 2
Win OS
Memory
Memory Authenticati
Protection on
Unix OS
Services
Processor Access Control
Modes

Files
User Secure Communication
Authentication (using cryptography)

File Access Logging &

Control Auditing

Intrusion
Prevention
and Detection

Recovery
 Operating System Overview
Operating system: collection
of programs that allows
user to operate computer
hardware

5
Operating System Overview
Key functions of an operating system:
1. Multitasking, multisharing
2. Computer resource management
3. Controls the flow of activities
4. Provides a user interface
5. Administers user actions and accounts
6. Runs software utilities and programs
7. Enforce security measures
8. Schedule jobs
6
 Security Goals of Operating Systems
1. Enable multiple users to securely share a
computer
Separation and sharing of processes, memory, files,
devices, etc.
2. Ensure secure operation in networked
environments
Enforce security policies while allowing resource
sharing across multiple computers

7
Goal 1: What Security Does Operating System
Provide?
Traditionally: enabling multiple users securely share a
computer
Separation and sharing of processes, memory, files, devices,
etc.
What is the threat model?
Users may be malicious, users have terminal access to
computers, software may be malicious/buggy, and so on
Security mechanisms for Goal 1
1. Memory Protection
2. Processor Modes
3. User Authentication
4. File Access Control
Goal 2: What Security Does Operating
System Provide?
Nowadays: ensure secure operation in networked
environment
What is the threat model? enemies control the network;
can communicate with computers
Security mechanisms for Goal 2

1. Authentication
2. Access Control
3. Secure Communication (using cryptography)
4. Logging & Auditing
5. Intrusion Prevention and Detection
6. Recovery
 Basis of OS Protection: Separation
Separation
Keep one users objects separate form other users
Possible separations provided by the OS
1. Physical separation
Use different physical objects (e.g., printers)
2. Temporal separation
Executed at different times
3. Logic separation
OS constrains programs access so that it cannot access objects
outsides its permitted domain
4. Cryptographic separation
Processes conceal their data and computation in such a way that they
are unintelligible to outsiders
Computer System Components
Hardware
Provides basic computing resources (CPU, memory, I/O
devices).
Operating system
Controls and coordinates the use of the hardware among the
various application programs.
Applications programs
Define the ways in which the system resources are used to
solve the computing problems of the users.
Users
E.g., people, machines, other computers.

11
Abstract View of System
Components
The OS Security Environment
a compromised OS can compromise a
database environment
Physically protect the computer running the
OS (padlocks, chain locks, guards, cameras)
Model:
Bank building (operating system)
Safe (database)
Money (data)

13
The OS Security Environment
The Components of an OS Security
Environment

Where we have to maintain C.I.A when the

information is within OS?
1. Memory
2. Services
3. Files

15
The Components of an OS Security
Environment

16
1. Memory
Hardware memory available on the system
Can be corrupted by badly written software
Two options:
Stop using the program
Apply a patch (service pack) to fix it
Can harm data integrity
Can potentially exploit data for illegal use
Memory Protection: Access Control to
Memory
Ensures that one users process cannot access others
memory
fence
relocation
base/bounds register
segmentation
paging

Operating system and user processes need to have
different privileges
High-level View of Kernel Space vs.
User Space

Process 1 Process 2 Process n

The Kernel

Hardware (disks, network interfaces, etc.)

 Kernel space vs User space
Part of the OS runs in the kernel model
known as the OS kernel
Can execute any instruction and access any memory locations
Examples: access hardware devices, enable and disable
interrupts, change privileged processor state, access memory
management units, modify registers for various descriptor
tables

20
 Kernel space vs User space
Other parts of the OS run in the user mode, including
service programs (daemon programs), user
applications, etc.
they run as processes
they form the user space (or the user land)
Access to memory is limited
Cannot execute some instructions
Transition from user mode to system mode must be done
through well defined call gates (system calls)
 System Calls
Guarded Gates from user mode into kernel
mode
Transfer control to predefined entry point in more
privileged code, using a special CPU instruction
(often an interruption)
Allow the more privileged code to specify where it
will be entered as well as important processor state
at the time of entry
The higher privileged code, by examining processor
state set by the less privileged code and/or its stack,
determines what is being requested and whether to
allow it.
Access control
A reference monitor mediates all access to resources
Principle: Complete mediation: control all accesses to resources

Reference
monitor
User
process access request ? Resource

policy
2. Services
Main component of operating system security
environment
Operating system core utilities used to gain access to
the OS and its features include
1. User authentication
2. Remote access
3. Administration tasks
4. Password policies

25
 Authentication Methods
Authentication:
Verifies user identity
Permits access to the operating system
Physical authentication:
Allows physical entrance to company property
Magnetic cards and biometric measures
Digital authentication: verifies user identity by
digital means

26
 Digital Authentication Mechanism

Digital certificates: digital passport that

identifies and verifies holder of
certificate
Digital token (security token):
Small electronic device
Displays a number unique to the token
holder; used with the holders PIN as a
password
Uses a different password each time
27
Digital Authentication Mechanism
Digital card:
Also known as a security card or smart card
Similar to a credit card; uses an electronic circuit
instead of a magnetic strip
Stores user identification information

28
Digital Authentication Mechanism
Kerberos:
Developed by MIT
Uses unique keys a.k.a. tickets for authentication
purposes
is a computer
network authentication protocol that works on the
basis of 'tickets' to allow nodes communicating
over a non-secure network to prove their identity
to one another in a secure manner.
Kerberos protocol messages are protected against
eavesdropping and replay attacks.
29
 Digital Authentication Mechanism
Lightweight Directory Access Protocol (LDAP):
Developed by the University of Michigan
A centralized directory database stores:
Users (user name and user ID)
Passwords
Internal telephone directory
Security keys
Efficient for reading but not suited for frequently
changing information
Easy to implement
Uses client/server architecture
Digital Authentication Mechanism
Public Key Infrastructure (PKI):
User keeps a private key
Authentication firm holds a public key
Encrypt and decrypt data using both keys

31
Digital Authentication Mechanism
RADIUS:
Remote authentication dial-in user services
used by network devices to provide a centralized
authentication mechanism
Secure Socket Layer (SSL): authentication information
is transmitted over the network in an encrypted form

Secure Remote Password (SRP):

Password is not stored locally
Invulnerable to brute force or dictionary attacks

32
Authorization

Process that decides whether users

are permitted to perform the
functions they request
Authorization is not performed until
the user is authenticated
Deals with privileges and rights

33
User Administration
Create user accounts
Set password policies
Grant privileges to users
Best practices:
Use a consistent naming convention
Always provide a password to an account and force
the user to change it at the first logon
Protect passwords
Do not use default passwords

34
User Administration (contd.)
Create a specific file system for users
Educate users on how to select a password
Lock non-used accounts
Grant privileges on a per host basis
Do not grant privileges to all machines
Use ssh, scp, and Secure FTP
Isolate a system after a compromise
Perform random auditing procedures
Password Policies
First line of defense
Dictionary attack: permutation of words in
dictionary
Make hard for hackers entering your systems
Best password policy:
Matches your company missions
Enforced at all level of the organization

36
Password Policies (contd.)
Best practices:

Password Aging
Password Reuse
Password History
Password Encryption

37
Password Policies (continued)
Best practices (continued):

Password Storage And Protection

Password Complexity
Logon Retries
Single Sign-On

38
3. Files
Common threats:
File permission
File sharing
Files must be protected from unauthorized reading
and writing actions
Data resides in files; protecting files
protects data
Read, write, and execute privileges

39
File Permission on Windows and Unix
 File Permission on Unix

Chmod 771 filename

File Transfer
FTP (File Transfer Protocol):
Internet service for transferring files from
one computer to another
Transmits usernames and passwords in
plaintext
Root account cannot be used with FTP
Anonymous FTP: ability to log on to the FTP
server without being authenticated

42
File Transfer (continued)
Best practices:
Use Secure FTP utility if possible
Make two FTP directories:
One for uploads with write permissions only
One for downloads with read permissions only
Use specific accounts with limited
permissions
Log and scan FTP activities
Allow only authorized operators

43
 Sharing Files
Naturally leads to security risks and threats
Peer-to-peer programs: allow users to share
files over the Internet
Reasons for blocking file sharing:
Malicious code
Adware and spyware
Privacy and confidentiality
Pornography
Copyright issues
Top vulnerabilities to Windows OS
1. Internet Information Services (IIS)

2. Microsoft SQL Server (MSSQL)

3. Windows Authentication

4. Internet Explorer (IE)

5. Windows Remote Access Services

6. Microsoft Data Access Components (MDAC)

7. Windows Scripting Host (WSH)

8. Microsoft Outlook and Outlook Express

9. Windows Peer-to-Peer File Sharing (P2P)

10. Simple Network Management Protocol (SNMP)

45
Top vulnerabilities to UNIX OS
1. BIND Domain Name System

2. Remote Procedure Calls (RPC)

3. Apache Web Server

4. General UNIX authentication accounts with no passwords or weak

passwords

5. Clear text services

6. Sendmail : Simple Network Management Protocol (SNMP)

7. Secure Shell (SSH)

8. Misconfiguration of Enterprise Services NIS/NFS

9. Open Secure Sockets Layer (SSL)

46
E-mail Security
Tool must widely used by public
May be the tool must frequently used by hackers:
Viruses
Worms
Spam
Others
Used to send private and confidential data as well as
offensive material

47
E-mail Security (continued)
Used by employees to communicate with:
Clients
Colleagues
Friends
Recommendations:
Do not configure e-mail server on the same machine where
sensitive data resides
Do not disclose technical details about the
e-mail server

48