You are on page 1of 36

Tracking Progress on

Security Awareness

Adrian Mikeliunas, CISSP, CISA


Tracking Progress on Security Awareness
AWR-1, November 6, 2006
Outline

• Security Awareness Purpose
• 4 Phases of Awareness
• Measuring Progress
• Case Studies
• Questions?

AWR­1 Tracking Progress on Security Awareness 2
Awareness Purpose

• Understand and comply with security 
policies and procedures
• Work to reduce errors and omissions by 
users due to lack of awareness and/or 
training
• 1st step in increasing Security 
– Awareness, Training, Education

AWR­1 Tracking Progress on Security Awareness 3
Four Phases of Security Awareness

Source NIST 800­50 

AWR­1 Tracking Progress on Security Awareness 4
Four Phases of Security Awareness

1. Design Project plan 
2. Develop or Purchase Security 
Awareness material
3. Implement program
• Pilot group
4. Post­implementation
• Measure results: Before & After

AWR­1 Tracking Progress on Security Awareness 5
Step 1 ­ Design

• Strategy & Goals
– Institutional long term view
– Conducting a Needs Assessment
• Prepare Training & Project plan
– Get feedback, observations
– Select Pilot Team
• Get management approval!
– Funding
– Agree on Benchmark statistics

AWR­1 Tracking Progress on Security Awareness 6
Step 1 ­ WIIFM

• Your motivation: 
– Keep your job!
– Obtain raise or promotion
– Raise Security Awareness!

WIIFM: What’s In It For Me?

AWR­1 Tracking Progress on Security Awareness 7
Source NIST 800­50 

AWR­1 Tracking Progress on Security Awareness 8
Step 2 – Obtain Material

• Develop 
• Purchase
• Outsource

• Test & Integrate 
– Learning Management System

• WIIFM: Quality, Relevant, Fun
AWR­1 Tracking Progress on Security Awareness 9
Step 3 ­ Implement program

• Pilot group
• Get feedback & support
• Adjust
• Involve population sample
• Savvy person
• Influential manager
• Contrarian

• WIIFM: Successful deployment
AWR­1 Tracking Progress on Security Awareness 10
Step 4 ­ Post­implementation

• Measure results: 
• Before 
• During
• After
• Report to Management on
• Identified Goals and Metrics
• Success can be measured by reduction of 
chronic problems, testing, surveying
• Leverage audit results, operational monitoring

• WIIFM: do it better next time!
AWR­1 Tracking Progress on Security Awareness 11
Success Indicators 

• 100% Compliance (or not!)
• Help desk calls/tickets
– Less password resets or data loss, 
– Less virus incidents
• Incident reports

AWR­1 Tracking Progress on Security Awareness 12
Monitoring Compliance

• Tracking compliance involves assessing 
the status of the program
• Reports to identify gaps or problems

AWR­1 Tracking Progress on Security Awareness 13
Example: US Government
• Total Number of Employees 4,222,251
• Employees that received IT security awareness training
• 3,427,756 or 81% 
• Total Number of Employees with significant IT security 
responsibilities 107,540
• Employees with significant responsibilities that received training 
88,939 or 83% 
• Total Costs for providing IT security training $79,389,201

• SOURCE: Government­wide Summary ­­ CIO Reports, 
OMB FY 2005 Report to Congress, 3/1/2006.

AWR­1 Tracking Progress on Security Awareness 14
Case 1

• Mandatory:
– Federal Law requires employees [or contractors] 
using, managing or operating Federal computer 
systems to receive annual IT Security Awareness 
and Training.
• 4 Main online modules
• Certificate for each completed section
• Feedback form at end of course

AWR­1 Tracking Progress on Security Awareness 15
AWR­1 Tracking Progress on Security Awareness 16
Case 1 – Training Completed

SEC IT Security Awareness and Training


Total Total
Completed Percent Population
SEC Total 4155 99.81% 4163
Government 3676 99.84% 3682
Contractor 478 99.38% 481

AWR­1 Tracking Progress on Security Awareness 17
Case 1 – Exceptions

• 68 ­ Users currently on the Training 
Exemption List which are not included in 
these totals:
– 53 of the 68 are New Users which are still 
within their 2 week grace period
– The remaining 15 are Medical, Regular 
Leave or Mission related exemptions

AWR­1 Tracking Progress on Security Awareness 18
Case 2 ­ Before

• Audit finding:
– Weaknesses in incident response, anti­virus, and 
password knowledge were revealed by the survey. 
– Most responders indicated that had not received 
any security awareness training in the past year
• Recommendation
– We recommend that security awareness training 
be conducted for each employee.

AWR­1 Tracking Progress on Security Awareness 19
Case 2 ­ Plan

• IFC is conducting a mandatory Computer 
Based Training (CBT) program based on 
BS7799 standards, customized for all IFC 
information users 
• IFC will continue to hold an annual 
Computer Security Day function to increase 
staff awareness
• IFC will host a national Peer Group 
Awareness Session for the Computer 
Security Institute.
AWR­1 Tracking Progress on Security Awareness 20
Case 2 – 
Awareness Pilot
• IFC has piloted a Computer Based Training 
(CBT) program based on BS7799 standards 
and customized for IFC.
• The pilot was given to a cross section of IFC 
information users (190+) at HQ & Country 
Offices.
• Based on the success of the pilot it was 
decided through the ISC to proceed with the 
awareness program to all IFC information 
users.
AWR­1 Tracking Progress on Security Awareness 21
Our Target Audiences

All Staff IT Professionals Senior Managers

 ‘Core’ e-learning ISSO e-learning Considering shorter


program courses initiatives focused
 Orientation for new Tailored to main job on specific needs of
hires families managers
 Ongoing initiatives
to maintain
awareness

AWR­1 Tracking Progress on Security Awareness 22
Deliverables: All Staff

For Your Eyes Only:


An e-learning program
covering all of our main IS
policy requirements
AWR­1 Tracking Progress on Security Awareness 23
E-Learning Pilot
Study Group
Abbreviated Items Ordered on Basis of Study
Time 1 Time 2
Group’s CHANGE Scores (from Time 1 to Time 2)

1. Encryption of confidential files and docs------------------------------------- + 39% 43% 82%

Percent Change in Perception of Importance (After Training)


2. Encryption of confidential e-mails-------------------------------------------- + 39% 46% 85%
3. Understanding how safe file downloading is done-------------------------- + 29% 61% 90%
4. Locking up confidential docs when leaving desk--------------------------- + 25% 66% 91%
5. Taking responsibility for keeping separated backups---------------------- + 22% 43% 65%
6. Recognition of illicit tactics to breach security----------------------------- + 20% 69% 89%
7. Security clearances-------------------------------------------------------------- + 18% 65% 83%
8. Non-disclosure agreements----------------------------------------------------- + 18% 39% 57%
9. Understanding proper password use------------------------------------------- + 11% 78% 89%
10. Mgrs' example on secure computer use--------------------------------------- + 11% 73% 84%
11. Increase resources to investigate security breaches------------------------- + 9% 53% 62%
12. Procedures for handling disasters---------------------------------------------- + 8% 80% 87%
13. Background checks on all staff------------------------------------------------ + 8% 52% 60%
14. Authentication method to verify system user IDs--------------------------- + 7% 78% 86%
15. Knowing how to prevent computer virus spread---------------------------- + 3% 89% 92%

AWR­1 Tracking Progress on Security Awareness 24
Deliverables: IT Professionals

E-Learning course covering BS7799


requirements (ISO 27001)
Related to needs of four job families:
Client Services
Information Management
Technology Management
Systems Analysis & Development

AWR­1 Tracking Progress on Security Awareness 25
Deliverables: Sr. Managers

• Senior managers and executives have 
specific training needs

• Some possible solutions may include:
– Executive briefing sessions in appropriate forums
– A video presentation of key issues
– A short, focussed e­learning program looking at 
organizational issues

AWR­1 Tracking Progress on Security Awareness 26
Case 2 ­Measure 
the effectiveness  T1
CBT Sessions
T2

• Baseline Pre­perception assessment 
(Time 1) to measure the current staff’s 
perception of key security issues
• CBT ­ Introduction of an independent 
variable, the Information Security 
Awareness Program
• Post­training perception assessment of 
Security Awareness after CBT  (Time 2)

AWR­1 Tracking Progress on Security Awareness 27
Population Sample
Experimental Group (N=237)

8
37

IFC HQ
36 IFC CO
156 WB HQ
WB CO

Control Group (N=87)

18

IFC HQ
11
IFC CO
58
WB HQ

AWR­1 Tracking Progress on Security Awareness 28
CBT followed by a second 
Questionnaire

• A sample of employees split into two 
groups, a study and a comparison 
group 
• The comparison group will control for 
other variables beyond the intended 
independent variable (the security 
awareness training)

AWR­1 Tracking Progress on Security Awareness 29
Before & After
Study Group Reaction to Questionnaire Study Group Reaction to CBT
at Time 1 (N=193) at Time 2 (N=193)
Response Percentage

40% 60%

Response Percentage
35%
30% 50%
25% 40%
20%
15% 30%
10% 20%
5%
0% 10%
Opened my eyes to potential Opened my eyes to existing problems 0%
problems Opened my eyes to potential problems Opened my eyes to existing problems
Questions Questions

Results:

CBT raised awareness by 20% for Question 1


and 15% for Question 2.

AWR­1 Tracking Progress on Security Awareness 30
Impact of CBT on Study Group
Study Group Abbreviated Item Content Rank Ordered on Basis of Study
Group’s CHANGE Scores
Time 1 Time 2

+41% Encryption of confidential files and docs


42% 82%
+40% Encryption of confidential e-mails
45% 85%
+29% Understanding how safe file downloading is done
61% 90%
+25% Locking up confidential docs when leaving desk
66% 91%
+22% Taking responsibility for keeping separated backups
43% 65%
+20% Recognition of illicit tactics to breach security
69% 89%
+18% Security clearances
65% 83%
+18% Non-disclosure agreements
39% 57%
+11% Understanding proper password use
78% 89%
+11% Mgrs' example on secure computer use
73% 84%
+9% Increase resources to investigate security breaches
53% 62%
+8% Procedures for handling disasters
80% 87%
+8% Background checks on all staff
52% 60%
+7% Authentication method to verify system user IDs
78% 86%
+3% Knowing how to prevent computer virus spread
89% 92%
AWR­1 Tracking Progress on Security Awareness 31
Distribution of Responses 
Across the 8 point scale

Distribution of Responses Across the 8 point scale


(for the Importance Questions numbers 7 - 21, see survey)

40.0%
Percentage

Study Group N=193


Response

30.0%
20.0% Comparison Group N=73
10.0%
0.0%
8 7 6 5 4 3 2 1
Response Categories
(8= Of Utmost Importance; 1=Somewhat Important)

AWR­1 Tracking Progress on Security Awareness 32
Change of Attitude Towards Security

Protection against unlikely security breaches &


Viruses should be considered more important than
Widespread, convenient data sharing

25%
20%
Percentage
Response

15% Time One


10% Time Two
5%
0%
1 2 3 4 5 6 7 8
Response Scale

AWR­1 Tracking Progress on Security Awareness 33
Change of Attitude Towards Monitoring

It is perfectly appropriate that employers develop and


utilize electronic methods to monitor how their
employees use their computers

25%
20%
Percentage
Response

15% Time One


10% Time Two
5%
0%
1 2 3 4 5 6 7 8
Response Scale

AWR­1 Tracking Progress on Security Awareness 34
0
5
10
15
20
25
30
Good CBT

CBT Content

Supervised

Mandatory

Data
Classification
Comments

CBT
Navigation

Encryption

AWR­1 Tracking Progress on Security Awareness
Password

Performance
What did staff tell us in the surveys?

F bldg.
design
35
Questions?
Email: Adrian@Mikeliunas.com

? ? ?
?
This is not the Beginning,
This is not the End,
But the End of the Beginning
AWR­1 Tracking Progress on Security Awareness 36