Component 2

The Certification of Graduates Using a National Competency-based Examination

Tri Hanggono Achmad
2nd HPEQ Conference Denpasar, 4 December 2011

Outline of presentation
Overview of Component 2 Report and key performance indicator Lesson learned Challenges and Future plan

Overview of Component 2

Health Professions Education Quality (HPEQ) Project

Rationale: Indonesias current health indicators, the current status of Indonesias health professional education programs, and on the increasing demand for larger numbers of and more highly qualified health professionals. Goals: strengthen quality assurance policies governing the education of health professionals in Indonesia through
1.

rationalizing and assuring competency focused accreditation of public and private health professional training institutions, developing national competency standards and testing procedures for certification and licensing of health professionals, and building institutional capacity to employ results based

2.

3.

Objectives
v

Sub-component 2.1: Establishing an

Independent National Agency for Competence Examination of Health Professionals

Sub-component 2.2: Improving the

Methodology and Management of the National Competency-based Examination

Sub-component 2.3: Developing an Item

Bank Networking System to Support the National Competence Examination

Main Program 2.1 National Agency

2010
kick off Paper-based, Preparation and Try out

2011
mid-term evaluation Implementation

2012
final evaluation

2013
take off

2014
sustainable

Implementatio Implementatio Implementatio n n n

2.2.a Computer-based Testing (CBT)

Paper-based, Paper-based, CBT CBT Implementatio Implementatio Implementatio Preparation and Preparation and n n n Try Out Try Out Preparation Preparation Preparation Preparation Benchmarking Test Try-out Try out Preparation Preparation Implementatio Implementatio and Try Out n n Implementatio Implementatio Implementatio n n n Implementatio Implementatio Implementatio n n n Preparation Try out Implementatio n

2.2.b Objective Structured Clinical Examination (OSCE)

Preparation

2.3 National Item Bank Networking for Assessment (NIBNA)

Enhancement Enhancement Enhancement for Pre-clinical for Clinical Establishment for Progress Formative Formative Test Test Test Enhancement Enhancement Enhancement for Pre-clinical for Pre-clinical for Clinical Establishment Formative Formative Formative Test Test Test Preparation Preparation Preparation Establishment

Preparation

Preparation

Medicine

Dentistry

Nursing/Midwi

HPEQ +
Pharmacy Nutrition Public Health Other health professionals related to MTKI

KPI PROJECT per YEAR

Key Performance Indicator Successfull rates on the first take of National Competence Examination
Health Baseline Professiona (2009) l Medicine Dental Nursing Midwifery Target Value 2010 2011 2012 2013 2014 84 % 83 % 65 % 65 %

71,67 % 71,67 % 73,25 % 75,50 % 79,60 % 79 % 80 % 80 % 81 % 82 % 60 % 60 %

Achievement of Component 2

Objectives
v

Sub-component 2.1: Establishing an

Independent National Agency for Competence Examination of Health Professionals

Sub-component 2.2: Improving the

Methodology and Management of the National Competency-based Examination

Sub-component 2.3: Developing an Item

Bank Networking System to Support the National Competence Examination

Subcomponent 2.1
Lembaga Pengembangan Uji Kompetensi (LPUK) An independent national agency that responsible in developing assessment strategy, methodology, and tools to evaluate competency of graduation from health professional education institution (medicine, dentist, nurse, midwife) Function:
1.

Assessment service: as resource

C h h a a r r

m m a a n n

E x e a c m u i National t n i a Regional v t e i Examination o Center s n e c D r i e v t i Tim Task Force 1300a/E3-HPEQ/SK/08.11 a s Director: Iwan Dwi Prahasto r i y o Secretary: Yulherina n Member:

Organization

Advisory board

The Milestone LPUK

Before 2010: First phase of development 2010 2014: Program Acceleration and Expantion After 2014: Sustainability Phase

Establishme nt of KBUKDI & KDGI Developmen t of KNUKP

Academic paper & various guidelines for workshop & implementati on of examination

Collaboratio n w/ other HP Resource Sharing International Benchmarkin g Human resource development Research grant & scientific publicati on Regional & International collaboratio n Recertification & assessment of professional behaviour

Various workshop in item development & review, standard setting Implementation of UKDI & UKDGI, and competence examination in some MTKPs

Resource & ICT support for assessment Establishme nt of LPUK & quality standard

Subcomponent 2.1
Benchmarking to National Board Examination for medicine, nurse & dentist Technical assistance from Prof. Gordon Page from Univ. British Columbia Canada Agreement with Ministry of Health
MoH Regulation Number 1796/Menkes/Per/VIII/2011 about Registration of health professional

Guidelines to conduct various workshop on assessment

Item development for CBT

Objectives
v

Sub-component 2.1: Establishing an

Independent National Agency for Competence Examination of Health Professionals

Sub-component 2.2: Improving the

Methodology and Management of the National Competency-based Examination

Sub-component 2.3: Developing an Item

Bank Networking System to Support the National Competence Examination

Subcomponent 2.2
Medicine & Dentist:
Standard setting with Modified Angoff Try out & CBT Implementation Try out OSCE

Nurse & Midwife:

Try out CBT in November 2011

Try out & CBT Implementation

Activity Medicine
Try Out 1 Try Out 2 Try Out 3 Try Out 4 Try Out 6 10 12 14 21

Number of CBT Center Dentist

1 6 11 13

Nurse
12 (809)

Midwife
13 (610)

Try Out OSCE

Medicin OSCE e Center Try Out 1 1 Try Out 12 2 Dentist OSCE Center Try Out 1 1 Try Out 13 2 Candidate s 14 160 Candidate s 9 274

Standard setting
Standard Number of Setting judges I II III IV V VI 52 52 46 48 50 66 Medicine Dentist Cut Score 49 49 Cut Score Number of judges 53 58 58 58 58 61 21 21

Judges: from health professions institutions Implementation of standard setting in some institutions

Number of CBT Center & Workstations

Total= 32 CBT Center Total workstations= 2892 PC/laptop (842 : 2140)

Average result of UKDI & UKDGI in 2010 2011

9.45 8.4 7.35 6.3 5.25 4.2 3.15 2.1 1.05 0 0.85 0.75 0.65 0.55 0.45 0.35 0.25 0.15 0.05 -0.05

Comparison of UKDI result based on Accreditation

ANOVA p=0,000

Objectives
v

Sub-component 2.1: Establishing an

Independent National Agency for Competence Examination of Health Professionals

Sub-component 2.2: Improving the

Methodology and Management of the National Competency-based Examination

Sub-component 2.3: Developing an Item

Bank Networking System to Support the National Competence Examination

Subcomponent 2.3 Establishment of NIBNA

Item development for knowledge based assessment CBT bank masing2 profesi Item development for skill based assessment OSCE Development of IT support system for student assessment Technical assistance for Item Bank Administration: Prof. Clarke Hazlet from Univ. Alberta Canada

Number of workshops & participants in 2011

No. Workshop 1 Frequency Item Review and Development (CBT)

Participant

Institution 65 (M) 26 (D) 168 (N) 129(Mid) 26 (M) 8 (D) 65 (M) 26 (D) 65 (M) 26 (D) 65 (M) 26 (D) 12 (M) 65 (M) 26 (D)

6 (4 M, 2 D) 389 (M) 8 (4 N, 4 Mid) 45 (D) 2 (other HP) 168 (N) 132 (Mid) 2 7 3 (2 M, 1 D) 3 (2 M, 1 D) 2 2 26 (M) 8 (D) 65 (M) 26 (D) 131 (M) 28 (D) 130 (M) 53 (D) 12 (M) 65 (M) 26 (D)

2 3 4 5 6 7

CBT Coordinator Standard Setting OSCE Examiner Item Review and Development (OSCE) OSCE Coordinator Standardized Patient

Medicine; D: Dentist; N: Nurse; Mid: Midwife; HP: Health Professions

Item Bank
Number of MCQ Item in Banks
12 10 8 6 4 2 0

OSCE: Medicine: 12 stations Dentist: 9 stations

Developing adequate quantity of high quality items: achievable target

Jumping Hurdles Successfully

If development is collaboratively produced & organized, realistic timelines for achieving a large-enough bank are - 5 yrs for Medicine (given 72 medical schools & 40,000 items reqd) - 10 yrs in Dentistry (if 26 dental schools & 30,000 (?) items reqd) - 4 yrs in Nursing (if 57 nursing schools & 25,000 (?) items reqd) - 3 yrs in Midwifery (if 60 midwifery schools & 20,000 (?) items reqd)

SIPENA Development Road Map

IT Modules interconnectivity
Formative Summative Call for Item Formative Summative Call for Item Sele Sele Sele Institution Assesment Institution Assesment Tes Tes Tes cted cted cted Bli N N N nd g in ted ted ted Ban Ban Ban Item Item Item Revie e e e Bank Bank Bank Ite Ite Ite w k k k Bank Bank Bank Bank Bank Bank Bl w : w : w in din 2: 2: 2: Re m m m : vie g Bank Bank Bank 1: 1: 1: w 3 3 3 It It It Item Writer / Item Writer / Item Writer / Revie Revie Revie Ban Ban Ban 4 4 4 Ne Ne Ne P : P : P : Teste Teste Teste e e e Reviewer Reviewer wed wed wed k k Reviewer k P P P Select Select Select w w w e e e d d d m m m Registration Registration Registration Item Item Item a a a Summative ed ed ed Ite Ite Ite Examinee Exam B e e e Item Item Exam Item B B Summative Examinee Exam Exam n Exami Exami n n - Scoring Item Item m Exami OL-Reg Location rm rItem Process rm Exami a a a - Scoring OL-Reg Location Process I I I el el el I nee Adm nk nk nee nee R R Local Local Local nk nee Local Adm R n E n n n National National NationalE National On On E On On e e Exam Exam Exam e P Exam P P P f f x f f xLine Exam Exam Exam x C Exam Line Line Line v C C v C Supervi Supervi Supervi v Supervi B o o p o o p B B B O O i O O Supervis Supervis Supervis Supervis Regist Regist Regist p B Regist iB i B B sor sor sor sor B B B B T T e T T r r r r S S or or e or or ration ration ration e T ration e S S e A A A A T T T Registr Registr Registr e A Registr A m m rt m m rt A A rt N N N N C C w C C Registrat Registrat Registrat Registrat w ss100722 w b b b b ation ation ation ation PadmaNet 2010 U U U U a a a a o o o o E E E E ion ion ion ion

Lesson learned
Building awareness of the role of assessment in improving the quality of life (health) and driving in prioritizing quality Support and Commitment from, and collaboration among various stakeholders More potencies being identified Resources sharing (human, infrastructure, items, system) efficiency & close the gap NCE support the improvement of assessment method & learning process in institutions assessment drives learning (standard of competence) Assessment as a learning / education materials and management process

Challenges & Future plan

The dynamic of the project continuity of the program based on the road map with possible modification Maintaing quality in bigger scope & quantity of project prioritizing & synchronize & coordination among Project Components & stakeholders resources sharing among health professionals Closing the gap of development & implementation paralel implementation of assessment development & learning response (case : OSCE & Clinical teaching) Customers friendly item banking & evidence based

Collaboration is Projects Mode of Operation

Since collaboration among health care educators also involves endorsement & participation of government & professional bodies in health care

This Project Can be & Will be Successful !

Terima Kasih
tachmad@yahoo.com spdewi@yahoo.com