18 EDUCATIONAL RESEARCH AND

DEVELOPMENT

OVERVIEW
This chapter describes a strategy for developing educational products of proven effectiveness. This
strategy is called research and development (R & D). ·It consists of a cycle in which a version of the
product is developed, field-tested, and revised on the basis of field-test data. Although product
development sometimes occurs in basic and applied research studies, their primary goal is to discover
new knowledge. In contrast, the goal of R&D is to take this research knowledge and incorporate it into
a product that can be used in the schools. In a sense, the purpose of R&D is to bridge the gap that
frequently exists between educational research and educational practice. The various steps of the R&D
cycle are described in this chapter as well as some of the problems and issues that confront developers
as they design a new product.

TINJAUAN
Bab ini menjelaskan strategi pengembangan produk pendidikan yang terbukti efektif. Strategi ini
disebut penelitian dan pengembangan (R & D). Ini terdiri dari sebuah siklus di mana versi produk
dikembangkan, diuji di lapangan, dan direvisi berdasarkan data uji lapangan. Meskipun pengembangan
produk kadang terjadi pada penelitian penelitian dasar dan terapan, tujuan utamanya adalah
menemukan pengetahuan baru. Sebaliknya, tujuan R & D adalah untuk mengambil pengetahuan
penelitian dan memasukkannya ke dalam produk yang dapat digunakan di sekolah. Dalam arti tertentu,
tujuan R & D adalah menjembatani kesenjangan yang sering terjadi antara penelitian pendidikan dan
praktik pendidikan. Berbagai langkah siklus R & D dijelaskan di bab ini dan juga beberapa masalah dan
masalah yang dihadapi pengembang saat merancang produk baru.

OBJECTIVES
After studying this chapter you should be able to:

1. State two deficiencies of basic and applied research as strategies for developing educational products.
2. Describe the 10 steps of the R&D cycle.
3. State the criteria that can be used to select an educational product to be developed.
4. Defend the importance of stating behavioral objectives in educational R&D.
5. Describe why it is important to field-test a product in a setting similar to that in which it will be used when fully
developed.
6. Explain the function of the main field test in the R&D cycle.
7. Give arguments for and against refinement of educational materials during the initial stages of development.
8. Describe two opportunities for a graduate student to do an R&D project.

TUJUAN
Setelah mempelajari bab ini, Anda seharusnya bisa:
1. Sebutkan dua kekurangan penelitian dasar dan terapan sebagai strategi untuk mengembangkan produk pendidikan.
2. Jelaskan 10 langkah siklus R & D.
3. Sebutkan kriteria yang bisa digunakan untuk memilih produk pendidikan yang akan dikembangkan.
4. Mempertahankan pentingnya menyatakan tujuan perilaku dalam R & D pendidikan.
 5. Jelaskan mengapa penting untuk menguji coba suatu produk dalam setting yang serupa dengan yang akan
digunakan saat dikembangkan sepenuhnya.
6. Jelaskan fungsi uji lapangan utama pada siklus R & D.
7. Berikan argumen untuk dan melawan penyempurnaan materi pendidikan selama tahap awal pengembangan.
8. Jelaskan dua kesempatan bagi mahasiswa pascasarjana untuk melakukan proyek Litbang.

WHAT IS EDUCATIONAL RESEARCH AND

DEVELOPMENT?
Research and development (R & D) is a powerful strategy for improving practice. Expenditures for R&D in
industry have averaged approximately 4 percent of sales during the past quarter century.l R&D expenditures in
some industries, such as pharmaceuticals and computer manufacture, are much higher than this. As a result,
technological advances in these fields have been rapid and have brought striking benefits for the consumer.
Unfortunately, R&D still plays a minor role in education. Less than one percent of education expenditures are for
2
this purpose. This is probably one of the main reasons why progress in education has lagged far behind progress
in other fields.
What is research and development (R & D)? It is a process used to develop and validate educational
products. By "product," we mean not only such things as textbooks, instructional films, and computer software,
but also methods, such as a method of teaching, and programs, such as a drug education program or a staff
development program. The focus of present-day R&D projects appears to be primarily on program development.
Programs are complex learning systems that often include specially developed materials and personnel trained to
work in a particular context.
The steps of the R&D process are usually referred to as the R&D cycle, which consists of studying research
findings pertinent to the product to be developed, deyeloping the product based on these findings, field testing it
in the setting where it will be used eventually, and revising it to correct the deficiencies found in the field-testing
stage. In more rigorous programs of R&D, this cycle is repeated until the field-test data indicate that the product
meets its behaviorally defined objectives.
In contrast, the goal of educational research is not to develop products, but rather to discover new
knowledge about fundamental phenomena (through basic research) or about educational practice (througn applied
research). Of course, many applied research projects involve development of educational products. For example,
in a project concerned with comparing the effectiveness of two methods for teaching reading, the researcher may
develop materials that incorporate each method because suitable materials are not available. Typically, however,
these materials are developed and refined only to the point where they can be used to test the researcher's
hypotheses. Furthermore, the tests are made in settings that do not reflect actual school conditions. For these
reasons applied research rarely yields products that are ready for operational use in the schools.
One way to bridge the gap between research and practice in education is to do R&D. It takes the findings
generated by basic and applied research and uses them to build tested products that are ready for operational use
in the schools. We should emphasize here, though, that educational R&D is not a substitute for basic or applied
research. All three research strategies – basic, applied, and R & D-are required to bring about educational change.
In fact, R&D increases the potential impact of basic and applied research findings upon school practice by
translating them into usable educational products.
Evaluation methods-especially formative and summative evaluation – play a major role in.R & D.
Therefore, if you plan to do an R&D study, you should study carefully the evaluation methods described in
chapter 17.
Educational R&D should not be equated with curriculum development .Curriculum development is often guided by
a curriculum phi1osophy or academic discipline rather than by the findings of empirical research. Also , the
development of curriculum guides and materials often does not involve field test and revision cycles. Some
curriculum developers, however, do use elements of R&D methodology in their work. As more of these elements
are used, , curriculum development approximates the R&D process.
Educational-R & D bears. a close relationship to the field of instructional technology. Instructional
technologists used to be concerned primarily with audiovisual equipment and materials, but in recent years their
primary interest has been the design and validation of learning systems. Reflecting this trend, we define
instructional technology as the systematic use of research knowledge and methods to design and validate learning
 systems. Many instructional technologists work in medical, military, and business settings where they primarily
develop training programs for employees.
If you plan to do an R&D project for your thesis or dissertation, we advise you to study instructional
technology to determine whether some of its methods and conceptual models are appropriate to your project.
These methods and models include: front - end analysis (needs assessment, system analysis, task analysis,
analysis of skill hierarchies); typologies of learning outcomes; match of instructional techniques to learning
outcomes; match of learner characteristics to instructional method; cognitive processes in learning ; individualized
instruction (Keller Plan, auto - tutorial instruction, mastery learning, etc.) ; and domam referenced assessment.
These methods and models are described in several textbooks mentioned in the Annotated References at the end
of this chapter. Also, it is useful to become acquainted with professional organizations of instructional
technologists and their publications: National society for Performance and Instruction (Performance and
Instruction Journal); Association for Educational Communications and Technology (Tech Trends; Educational
Communications and Technology Journal ), and American Society for Training and development (training and
Development Journal).

APA PENELITIAN DAN PEMBANGUNAN PENDIDIKAN?

Penelitian dan pengembangan (R & D) adalah strategi yang ampuh untuk memperbaiki praktik. Pengeluaran
untuk Litbang di industri memiliki rata-rata sekitar 4 persen dari penjualan selama seperempat abad yang lalu.
Pengeluaran Litbang di beberapa industri, seperti farmasi dan pembuatan komputer, jauh lebih tinggi daripada ini.
Akibatnya, kemajuan teknologi di bidang ini sangat pesat dan membawa manfaat mencolok bagi konsumen.
Sayangnya, R & D masih memainkan peran kecil dalam pendidikan. Kurang dari satu persen pengeluaran
pendidikan untuk tujuan ini.2 Ini mungkin salah satu alasan utama mengapa kemajuan dalam pendidikan
tertinggal jauh dari kemajuan di bidang lain.
Apa itu penelitian dan pengembangan (Litbang)? Ini adalah proses yang digunakan untuk mengembangkan
dan memvalidasi produk pendidikan. Dengan "produk", kami bermaksud tidak hanya hal-hal seperti buku teks,
film instruksional, dan perangkat lunak komputer, tetapi juga metode, seperti metode pengajaran, dan program,
seperti program pendidikan obat atau program pengembangan staf. Fokus proyek R & D sekarang tampaknya
terutama pada pengembangan program. Program adalah sistem pembelajaran yang kompleks yang seringkali
mencakup bahan dan personil yang dikembangkan secara khusus yang terlatih untuk bekerja dalam konteks
tertentu.
Langkah-langkah proses Litbang biasanya disebut sebagai siklus Litbang, yang terdiri dari mempelajari
temuan penelitian yang berkaitan dengan produk yang akan dikembangkan, mengidentifikasi produk berdasarkan
temuan ini, menguji lapangan di tempat di mana nantinya akan digunakan, dan merevisinya untuk memperbaiki
kekurangan yang ditemukan di tahap pengujian lapangan. Dalam program R & D yang lebih ketat, siklus ini
diulangi sampai data uji lapangan menunjukkan bahwa produk tersebut memenuhi tujuan yang ditentukan secara
perilaku.
Sebaliknya, tujuan penelitian pendidikan bukan untuk mengembangkan produk, melainkan untuk
menemukan pengetahuan baru tentang fenomena fundamental (melalui penelitian dasar) atau tentang praktik
pendidikan (penelitian terapan). Tentu saja, banyak proyek penelitian terapan melibatkan pengembangan produk
pendidikan. Misalnya, dalam sebuah proyek yang berkaitan dengan membandingkan keefektifan dua metode
untuk pengajaran membaca, peneliti dapat mengembangkan materi yang menggabungkan setiap metode karena
bahan yang sesuai tidak tersedia. Biasanya, bagaimanapun, materi ini dikembangkan dan disempurnakan hanya
sampai pada titik di mana mereka dapat digunakan untuk menguji hipotesis peneliti. Selanjutnya, tes dilakukan di
setting yang tidak mencerminkan kondisi sekolah yang sebenarnya. Untuk alasan ini penelitian terapan jarang
menghasilkan produk yang siap digunakan operasional di sekolah.
Salah satu cara untuk menjembatani kesenjangan antara penelitian dan praktik di bidang pendidikan adalah
melakukan litbang. Diperlukan temuan yang dihasilkan oleh penelitian dasar dan terapan dan menggunakannya
untuk membangun produk teruji yang siap digunakan operasional di sekolah. Namun, kita harus menekankan di
sini bahwa R & D pendidikan tidak menjadi pengganti riset dasar atau terapan. Ketiga strategi penelitian - dasar,
terapan, dan litbang - dibutuhkan untuk mendatangkan perubahan pendidikan. Faktanya, R & D meningkatkan
potensi dampak temuan penelitian dasar dan terapan pada praktik sekolah dengan menerjemahkannya menjadi
produk pendidikan yang dapat digunakan.
 Metode evaluasi - terutama evaluasi formatif dan sumatif - memainkan peran utama dalam. D. Oleh karena
itu, jika Anda berencana melakukan studi Litbang, Anda harus mempelajari dengan seksama metode evaluasi
yang dijelaskan dalam bab 17.
R & D pendidikan tidak boleh disamakan dengan pengembangan kurikulum. Pengembangan kurikulum
sering dipandu oleh filosofi kurikulum atau disiplin akademis dan bukan oleh temuan penelitian empiris. Juga,
pengembangan panduan dan materi kurikulum seringkali tidak melibatkan uji lapangan dan siklus revisi.
Beberapa pengembang kurikulum, bagaimanapun, menggunakan elemen metodologi R & D dalam pekerjaan
mereka. Karena lebih banyak elemen ini digunakan,, pengembangan kurikulum mendekati proses Litbang.
Beruang Pendidikan-R & D. hubungan erat dengan bidang teknologi instruksional. Teknisi instruksional
dulu terutama memperhatikan peralatan dan bahan audiovisual, namun dalam beberapa tahun terakhir, minat
utama mereka adalah desain dan validasi sistem pembelajaran. Mencerminkan kecenderungan ini, kita
mendefinisikan teknologi instruksional sebagai penggunaan pengetahuan dan metode penelitian yang sistematis
untuk merancang dan memvalidasi sistem pembelajaran. Banyak teknolog instruksional bekerja di lingkungan
medis, militer, dan bisnis dimana mereka terutama mengembangkan program pelatihan untuk karyawan.
Jika Anda berencana untuk melakukan proyek Litbang untuk tesis atau disertasi Anda, kami menyarankan
Anda untuk mempelajari teknologi instruksional untuk menentukan apakah beberapa metode dan model
konseptualnya sesuai untuk proyek Anda. Metode dan model ini meliputi: analisis front - end (penilaian
kebutuhan, analisis sistem, analisis tugas, analisis hierarki keterampilan); tipologi hasil belajar; kecocokan teknik
instruksional terhadap hasil belajar; sesuai dengan karakteristik peserta didik dengan metode instruksional; proses
kognitif dalam belajar; instruksi individual (Keller Plan, instruksi tutorial otomatis, pembelajaran penguasaan,
dll.); dan penilaian yang diacu oleh domita. Metode dan model ini dijelaskan dalam beberapa buku teks yang
disebutkan dalam Acuan Beranotasi di akhir bab ini. Selain itu, juga berguna untuk mengenal organisasi
profesional teknolog instruksional dan publikasi mereka: Masyarakat Nasional untuk Kinerja dan Instruksi (Jurnal
Kinerja dan Instruksi); Asosiasi Komunikasi dan Teknologi Pendidikan (Tech Trends, Jurnal Pendidikan
Komunikasi dan Teknologi), dan American Society for Training and Development (Jurnal Pelatihan dan
Pengembangan).

THE R&D CYCLE

In the remainder of this chapter we shall discuss each of the major steps in the R & D cycle. The specific
R&D cycle that will be presented was developed by the staff of the Teacher Education Program at the Far West
Laboratory for Educational Research and Development, with which the authors were formerly affiliated. The Far
West Laboratory is one of 10 regional laboratories funded by the U.S. Office of Education to bring about
educational improvement through R & D. The Teacher Education Program developed products called
3
minicourses, which were designed to improve teachers' use of specific classroom skills.
Since we will be using the development of our first minicourse to illustrate the R&D cycle, we will briefly
describe here the characteristics of this product. Each minicourse involves about 15 hours of teacher training in
either the preservice or inservice setting. During this time, the teacher being trained is introduced to a number of
specific classroom skills. These skills are first described and illustrated in an instructional film. The teacher then
sees the skills demonstrated in a "model film," that is, a film of a brief classroom situation conducted by a model
teacher. Then the teacher plans a short lesson in which the skills can be applied, teaches the lesson to a small
group of students, and records the lesson on videotape. Immediately after the lesson, the teacher views the
videotape, focusing attention on the specific skills to be learned.
This lesson is called a microteach lesson because the regular classroom situation is scaled down in time and
number of pupils. Having seen and evaluated the videotape recording of the lesson, the teacher then replans the
same lesson and reteaches it the following day to another small group of pupils. This lesson is also recorded on
videotape, so that the teacher can again view and evaluate his or her performance immediately after the lesson is
completed. The teacher then proceeds to the next sequence of instructional lesson, model lesson, microteach, and
reteach.
The major steps in the R&D cycle used to develop minicourses are as follows:
1. Research and information collecting-Includes needs assessment, review of literature, small-scale research
studies, and preparation of report on state of the art.
2. Planning-Includes defining skills to be learned, stating and sequencing objectives, identifying learning
activities, and small-scale feasibility testing.
3. Develop preliminary form of product-Includes preparation of instructional materials, procedures, and evaluation
instruments.
4. Preliminary field testing-Conducted in from 1 to 3 schools, using 6 to 12 subjects. Interview, observational, and
questionnaire data collected and analyzed.
5. Main product revision-Revision of product as suggested by the preliminary field-test results.
6. Main field testing-Conducted in 5 to 15 schools with 30 to 100 subjects. Quantitative data on subjects' precourse
and postcourse performance are collected. Results are evaluated with respect to course objectives and are
compared with control group data, when appropriate.
7. Operational product revision-Revision of product as suggested by main field-test results.
8. Operational field testing-Conducted in 10 to 30 schools involving 40 to 200 subjects. Interview, observational,
and questionnaire data collected and analyzed.
9. Final product revision-Revision of product as suggested by operational field-test results.
10. Dissemination and implementation-Report on product at professional meetings and in journals. Work with
publisher who assumes commercial distribution. Monitor distribution to provide quality control.

If this sequence of 10 steps is followed properly, it yields a validated educational product that is fully ready for
operational use in the schools. Each of the steps is described below. Some of the steps described above, especially
step 6 (main field testing), involve research methods described in previous chapters.

SIKLUS R & D
Dalam sisa bab ini kita akan membahas setiap langkah utama dalam siklus R & D. Siklus litbang yang spesifik yang
akan dipresentasikan dikembangkan oleh staf Program Pendidikan Guru di Laboratorium Far East untuk Penelitian
dan Pengembangan Pendidikan, dimana penulis tersebut sebelumnya berafiliasi. Laboratorium Far West adalah satu
dari 10 laboratorium regional yang didanai oleh Dinas Pendidikan A.S. untuk mewujudkan peningkatan pendidikan
melalui R & D. Program Pendidikan Guru mengembangkan produk yang disebut minicourses, yang dirancang untuk
meningkatkan kemampuan kelas guru dalam keterampilan kelas tertentu.
Karena kita akan menggunakan pengembangan minicourse pertama kita untuk menggambarkan siklus R & D, kita
akan menjelaskan secara singkat karakteristik produk ini di sini. Setiap minicourse melibatkan sekitar 15 jam
pelatihan guru baik di lingkungan preservice maupun inservice. Selama masa ini, guru yang dilatih diperkenalkan
pada sejumlah keterampilan kelas tertentu. Keterampilan ini pertama kali dijelaskan dan diilustrasikan dalam sebuah
film instruksional. Guru kemudian melihat keterampilan yang ditunjukkan dalam "film model", yaitu sebuah film
situasi kelas singkat yang dilakukan oleh seorang guru model. Kemudian guru merencanakan sebuah pelajaran
singkat di mana keterampilan dapat diterapkan, mengajarkan pelajaran kepada sekelompok kecil siswa, dan
mencatat pelajaran tentang rekaman video. Segera setelah pelajaran, guru melihat rekaman video, memusatkan
perhatian pada keterampilan spesifik untuk dipelajari.
Pelajaran ini disebut pelajaran microteach karena situasi kelas reguler diperkecil dalam waktu dan jumlah murid.
Setelah melihat dan mengevaluasi rekaman rekaman video pelajaran, guru kemudian mengganti pelajaran yang
sama dan mengulanginya keesokan harinya ke kelompok murid kecil lainnya. Pelajaran ini juga direkam dalam
rekaman video, sehingga guru dapat kembali melihat dan mengevaluasi kinerjanya segera setelah pelajaran selesai.
Guru kemudian melanjutkan ke urutan pelajaran instruksional berikutnya, pelajaran model, microteach, dan reteach.
Langkah utama dalam siklus R & D yang digunakan untuk mengembangkan minicourses adalah sebagai berikut:
1. Pengumpulan informasi dan penelitian-Meliputi penilaian kebutuhan, tinjauan literatur, penelitian skala kecil, dan
penyusunan laporan mutakhir.
2. Perencanaan-Meliputi ketrampilan menentukan untuk dipelajari, menyatakan dan menetapkan tujuan,
mengidentifikasi kegiatan pembelajaran, dan uji kelayakan skala kecil.
3. Mengembangkan bentuk awal produk-Termasuk persiapan bahan ajar, prosedur, dan instrumen evaluasi.
4. Pengujian lapangan awal-Dilakukan di 1 sampai 3 sekolah, menggunakan 6 sampai 12 subjek. Data wawancara,
observasional, dan kuesioner dikumpulkan dan dianalisis.
5. Revisi produk utama-Revisi produk seperti yang disarankan oleh hasil uji lapangan awal.
6. Uji lapangan utama-Dilakukan di 5 sampai 15 sekolah dengan 30 sampai 100 subjek. Data kuantitatif tentang
prakiraan subyek dan kinerja postcourse dikumpulkan. Hasil dievaluasi sehubungan dengan tujuan kursus dan
dibandingkan dengan data kelompok kontrol, jika sesuai.
7. Revisi produk operasional - Revisi produk seperti yang disarankan oleh hasil uji lapangan utama.
8. Uji lapangan operasional-Dilakukan di 10 sampai 30 sekolah yang melibatkan 40 sampai 200 subjek. Data
wawancara, observasional, dan kuesioner dikumpulkan dan dianalisis.
9. Revisi produk akhir - Revisi produk seperti yang disarankan oleh hasil uji lapangan operasional.
10. Penyebarluasan dan implementasi - Melaporkan produk pada pertemuan profesional dan dalam jurnal.
Bekerjalah dengan penerbit yang menganggap distribusi komersial. Monitor distribusi untuk memberikan kontrol
kualitas.

Jika urutan 10 langkah ini diikuti dengan benar, ini menghasilkan produk pendidikan yang divalidasi sepenuhnya
yang siap digunakan di sekolah. Masing-masing langkah dijelaskan di bawah ini. Beberapa langkah yang dijelaskan
di atas, terutama langkah 6 (pengujian lapangan utama), melibatkan metode penelitian yang dijelaskan di bab
sebelumnya.

Research and Information Collection

Needs Assessment
Several criteria should be considered in selecting a product for development. The criteria
used at the Far West Laboratory included the following:

1. Does the proposed product meet an important educational need?

2. Is the state of the art sufficiently advanced that there is a reasonable probability
that a successful product can be built?
3. Are personnel available who have the skills, knowledge, and experience
necessary to build this product?
4. Can the product be developed within a reasonable period of time?

The first criterion – need - can be addressed by doing a needs assessment. Procedures that
an independent developer might use to assess needs are described in chapter 17.
Procedures that are used by educational R&D organizations to assess needs may take a
different form.4 For example, they often must address needs that are stated in funding
proposals, irrespective of whether the target audience perceives these needs.
It was apparent to the staff of the Teacher Education Program that there was a pressing
need to develop effective products for inservice teacher eflucation. School districts at that
time generally provided very little inservice education, and what was available suffered
from four serious weaknesses: (1) Teachers were told what to do rather than being given
the opportunity to practice good teaching techniques; (2) they were taught vague
generalities, such as "individualize your instruction," rather than being given training in
specific classroom skills; (3) they were not shown effective models to emulate; and (4)
they were given little or no feedback on their classroom performance. The minicourses
were designed to overcome these weaknesses.

Koleksi Penelitian dan Informasi

Butuh penilaian
Beberapa kriteria harus dipertimbangkan dalam memilih produk untuk pengembangan.
Kriteria yang digunakan di Laboratorium Far West meliputi:
1. Apakah produk yang diusulkan memenuhi kebutuhan pendidikan yang penting?
2. Apakah keadaan seni cukup maju sehingga ada kemungkinan yang masuk akal bahwa
produk yang sukses dapat dibangun?
3. Apakah personil tersedia yang memiliki keterampilan, pengetahuan, dan pengalaman
yang diperlukan untuk membangun produk ini?
4. Bisakah produk dikembangkan dalam jangka waktu yang wajar?
 Kriteria pertama - kebutuhan - dapat diatasi dengan melakukan penilaian kebutuhan.
Prosedur yang dapat digunakan oleh pengembang independen untuk menilai kebutuhan
dijelaskan di Bab 17. Prosedur yang digunakan oleh organisasi litbang pendidikan untuk
menilai kebutuhan dapat mengambil bentuk yang berbeda.4 Misalnya, mereka seringkali
harus memenuhi kebutuhan yang tercantum dalam proposal pendanaan, terlepas dari
apakah target pemirsa merasakan kebutuhan ini?
Hal ini terlihat jelas bagi staf Program Pendidikan Guru bahwa ada kebutuhan mendesak
untuk mengembangkan produk yang efektif untuk pengembangan guru inservice. Distrik
sekolah pada waktu itu umumnya menyediakan pendidikan inservice yang sangat sedikit,
dan apa yang tersedia menderita empat kelemahan serius: (1) Guru diberi tahu apa yang
harus dilakukan daripada diberi kesempatan untuk mempraktikkan teknik pengajaran
yang baik; (2) mereka diajarkan generalisasi kabur, seperti "individualalize your
instruction," daripada diberi pelatihan dalam keterampilan kelas tertentu; (3) mereka
tidak menunjukkan model yang efektif untuk ditiru; dan (4) mereka diberi sedikit atau
tidak ada umpan balik mengenai kinerja kelas mereka. Minikour dirancang untuk
mengatasi kelemahan ini.

Literature Review
Once the nature of the educational product has been tentatively identified, a literature
review is undertaken to collect research findings and other information pertinent to the
planned development. As in basic or applied research, one purpose of the literature
review is to determine the state of knowledge in the area of concern. In R&D projects,
the developer also must be concerned with how this knowledge can be applied to the
planned product.
A prelimmary review of the literature on teaching methods suggested that
questioning techniques in classroom discussions would be a good choice for our first
minicourse. The title eventually given to Minicourse 1 was "Effective Questioning -
Elementary Level."
Since Minicourse 1 was the first product developed by the Teacher Education
Program, it was necessary to conduct two literature reviews. The purpose of the first
review was to locate research that could be used to develop a basic instructional moder
for training teachers. Research in four areas was studied : microteaching, Iearning from
films, feedback in learning, and modeling in learning. Through this review we were able
to identify several instructional techniques that improve learning. For example, research
has found that providing teachers with videotape feedback on their teaching performance
is an effective technique for developing new classroom skills. Another effective
technique is to provide a model of the skills to be learned.
Our second literature review was concerned with questioning and discussion
skills.We found that research in this area : extended back to Stevens's 1912 study of high
school classrooms.5 Stevens found that two-thirds of teachers' questions required students
to recall facts rather than to think about facts. Furthermore, teachers talked two-thirds of
the discussion time, thus allowing students to participate only one-third of the time.
6
Similar findings have been obtained in more recent studies. It appears that even though
they have known about the prevalence of such underiable teaching practices for a long
time, educators have not succeeded in bringing about needed improvements in teachers'
classrom skills. We decided that major goals of Minicourse 1 would be to reduce teacher
talk and correspondingly to increase student talk, and to increase the percentage of
teachers' thought questions.
In the next phase of the literature review, it was necessary to identify specific
techniques that teacher could use to accomplish these goals. Although some pertinent
 research studies were available, it was also necessary for us to give considerable attention
to the opinions and experience of practitioners. For example, Grossier advocates several
teaching strategies which were included in Minicourse 1, but he presents no evidence on
7
their effectiveness. Since our later field experience with Minicourse 1 indicated that most
of the strategies bring about improved class discussion, they were included in the final
form of the course.

Tinjauan Literatur

Begitu sifat produk pendidikan telah diidentifikasi sementara, tinjauan literatur

dilakukan untuk mengumpulkan temuan penelitian dan informasi lain yang berkaitan
dengan rencana pembangunan. Seperti pada penelitian dasar atau terapan, salah satu
tujuan kajian literatur adalah untuk mengetahui keadaan pengetahuan di bidang yang
menjadi perhatian. Dalam proyek Litbang, pengembang juga harus memperhatikan
bagaimana pengetahuan ini dapat diterapkan pada produk yang direncanakan.

Sebuah tinjauan prelek dari literatur tentang metode pengajaran menunjukkan

bahwa teknik tanya jawab dalam diskusi kelas akan menjadi pilihan yang baik untuk
minicourse pertama kita. Judul yang diberikan kepada Minicourse 1 adalah "Effective
Questioning - Elementary Level."

Karena Minicourse 1 adalah produk pertama yang dikembangkan oleh Program

Pendidikan Guru, perlu dilakukan dua ulasan literatur. Tujuan dari tinjauan pertama
adalah untuk menemukan penelitian yang dapat digunakan untuk mengembangkan
bahasa instruksional dasar untuk melatih guru. Penelitian di empat bidang tersebut
dipelajari: microteaching, pembelajaran dari film, umpan balik dalam pembelajaran, dan
pemodelan dalam pembelajaran. Melalui review ini kami dapat mengidentifikasi
beberapa teknik pembelajaran yang meningkatkan pembelajaran. Misalnya, penelitian
telah menemukan bahwa memberi umpan balik kepada timbal balik kepada siswa tentang
kinerja pengajaran mereka adalah teknik yang efektif untuk mengembangkan
keterampilan kelas baru. Teknik lain yang efektif adalah dengan memberikan model
ketrampilan untuk dipelajari.

Tinjauan literatur kedua kami terkait dengan keterampilan mempertanyakan dan

diskusi. Kami menemukan bahwa penelitian di bidang ini: diperpanjang kembali ke studi
tahun 1912 di kelas lanjutan di Harvard.1 Stevens menemukan bahwa dua pertiga
pertanyaan guru mengharuskan siswa untuk mengingat fakta dan bukan pada pikirkan
fakta Selanjutnya, guru berbicara dua pertiga dari waktu diskusi, sehingga
memungkinkan siswa untuk berpartisipasi hanya sepertiga dari waktu. Temuan serupa
telah diperoleh dalam penelitian yang lebih baru.6 Tampaknya meskipun mereka telah
mengetahui tentang prevalensi praktik pengajaran yang tidak memadai tersebut sejak
lama, para pendidik tidak berhasil membawa perbaikan keterampilan kelas guru yang
diperlukan. Kami memutuskan bahwa tujuan utama Minicourse 1 adalah mengurangi
obrolan guru dan untuk meningkatkan pembicaraan siswa, dan untuk meningkatkan
persentase pertanyaan pemikiran guru.

Pada tahap selanjutnya dari tinjauan literatur, perlu untuk mengidentifikasi teknik
spesifik yang dapat digunakan guru untuk mencapai tujuan ini. Meskipun ada beberapa
 penelitian penting yang tersedia, kami juga perlu memberi perhatian besar pada pendapat
dan pengalaman praktisi. Misalnya, Grossier menganjurkan beberapa strategi pengajaran
yang termasuk dalam Minicourse 1, namun dia tidak menemukan bukti efektivitasnya.7
Sejak pengalaman lapangan kami di Minicourse 1 menunjukkan bahwa sebagian besar
strategi tersebut menghasilkan diskusi kelas yang lebih baik, mereka disertakan dalam
bentuk akhir kursus.

Small-Scale Research
Developers often will have questions that cannot be answered by referring to research
studies or professional texts. These questions can be answered at least tentatively by
doing small-scale studies prior to developing the product. For example, in Minicourse 5,
which is concerned with mathematics tutoring skills, we could find no research findings
about what occurs between pupil and teacher in the typical tutoring sequence. In order to
partially fill this gap, we sent observers into a number of classrooms to study tutoring
interactions between teachers and pupils. We learned from these observations that the
usual tutoring contact between the teacher and the individual pupil was brief, averaging
only 15 seconds. The content of these tutoring contacts suggested that the teacher
typically gave the pupil an answer or pointed out the pupil's error and then moved on.
Efforts to guide the pupil toward the identification of his or her errors or to develop
understanding of mathematical concepts and problem-solving procedures were rare.
Although they were not collected in a tightly controlled research setting, these data gave
us at least a tentative empirical basis for determining a direction in which to improve
teachers' tutoring skills.

Penelitian Skala Kecil

Pengembang sering akan memiliki pertanyaan yang tidak bisa dijawab dengan mengacu
pada studi penelitian atau teks profesional. Pertanyaan-pertanyaan ini dapat dijawab
setidaknya secara tentatif dengan melakukan studi skala kecil sebelum mengembangkan
produk. Misalnya, di Minicourse 5, yang peduli dengan keterampilan les matematika, kita
tidak dapat menemukan temuan penelitian tentang apa yang terjadi antara murid dan guru
dalam urutan les yang khas. Untuk mengisi sebagian kesenjangan ini, kami mengirim
pengamat ke sejumlah ruang kelas untuk belajar mempelajari interaksi antara guru dan
murid. Kami belajar dari pengamatan ini bahwa kontak les biasa antara guru dan murid
masing-masing singkat, rata-rata hanya 15 detik. Isi dari kontak les ini menunjukkan
bahwa guru biasanya memberi jawaban pada murid atau menunjukkan kesalahan murid
dan kemudian melanjutkan. Upaya untuk membimbing murid menuju identifikasi
kesalahannya atau untuk mengembangkan pemahaman tentang konsep matematika dan
prosedur pemecahan masalah sangat jarang terjadi. Meskipun mereka tidak dikumpulkan
dalam pengaturan penelitian yang dikontrol ketat, data ini memberi kami setidaknya
dasar empiris untuk menentukan arah di mana untuk meningkatkan keterampilan
bimbingan guru.

Planning
Once the developer has completed a literature review and collection of other pertinent
information, the next step is to make a plan of the product. This plan should include: (1) the
 product’s objectives, (2) the product’s target audience, and (3) a description of the product's
components and how they will be used.
The product often changes substantially during the development process. This does not
mean that the initial planning should be taken lightly. It providess
the information upon which later revisions are built. Without carefulpJanninltat the start, the likelihood
9f buildin a ood roduct ir much reduced.
The most important aspect of the plan is the statem nt . es to be achit;ved by the Eroduct. For
example, an objective of an R&D product in social studies might be, ~st75 percent of the students who
1/

c~tethe pr~ramwill earn a score of 90 or better on a test measuring various map skills." Such student-
based objectives enable educators to determine in quantitative terms whether the program "works."
Objectives also provide the best basis for developing an instructional program, because the program
can be field-tested and revised until it meets its objectives. Precise specification of educational
outcomes-or behavioral objectives, as they are also called-requires considerable skill on the part of the
s
developer. In some ways developing a behavioral objective for an educational product is similar to
developing a good criterion in a research study.
During the planning phase, behavioral objectives are usually stated somewhat loosely. For
example, in the initial planning of Minicourse I, one of our objectives stated that after the course most
t woul' ea . use of thou ht 'ons in a dis . . i~. We did not have sufficient knowledge in the planning
phase, though, to sp"ecify the petcentage of thought questions that we would expect teachers to ask in
order for the course to be considered effective. As we proceeded through the R&D cycle and
accumulated research data, we were able to refine the statement of the behavioral objective so that it
took the following form: "Given a 20-minute discussion lesson, at least half of all questions asked by
teachers will be classified as thought questions. This criterion will be met by at least 75 percent of
teachers who complete Minicourse 1."

r Another important element of the planning phase is estimation of the money, personnel, and

time required to develop tbe product. Generally, ample resources are needed to carry out a single R&D
project. The cost of developing a single minicourse, which provides about 15 hours of instruction, was
in excess of $100,000 in the early 1970s. A major curriculum project today will cost several

...) ~ .. ,.. ".million dollars. Personnel needs are considerable, too. The d~velopment of a
~';iimcourse requires an average of 104 person-weeks of professional work, 50 ) person-weeks of clerical work,
and 50 person-weeks of production work. In '" \ contrast, most research projects involve small sums of money, often
just a thousand dollars or so, and the efforts of a single investigator with perhaps a few part-time graduate assistants.
Unless careful planning is done, developers may find that their resources have run out before the product has been
fully developed, PJan~ngis ~€!.£essal;.y in order to anticipat~ed materials, profe~sional help" and field-test..§ires.
ConSIderation of field-test sites is especially im~wh-entesting is done in

the schools, which generally are receptive to testing only at certain times of the year. For'example,
if the product is ready for testing in June, one may have to wait until September or October unless
the product can be tested during a summer school session. Also, school administrators generally
require a few months' notice before agreeing to have their schools serve as a test site.
Although developers must devote a considerable amount of time to initial planning, the
planning function is never really ended. As work progresses, they are likely to discover
several areas in which initial planning was insufficient or in error. Replanning must then be
done. Nonetheless, it is wise to devote major effort to building a sound initial plan. A~ood plan
can help d.~elo~rs avoi,d much wasted work during later phases of the R&D cycle.
~.....
Development of the Preliminary Form of the Product
After the initial planning has been completed, the next major step in the R&D cycle is to
build a preliminary form of the educational roduct that can b field tested. In the case
'OfNfmIcourse 1, thIS mvolved a wide range 0 tasks. Scripts describing the specific skills that
teachers are to learn were written lor each instructional sequence-:T'ne scripts werethen
produced on videotape and edited to include clips showing the skills being used in classroom
situations. Prospective model teachers were located, observed, and trained to conduct model
lessons designed to further illustrate the minicourse skills. The model lessons were then
recorded on videotape and edited. A teacher handbook designed to supplement the videotaped
instructional lessons was drafted, revised, and printed. A set of forms for teachers to use in
self-evaluation of the microteach and reteach lessons was developed and printed.
An important principle to follow at this stage of proj.llCt...ae-ve-I.opmenU$-.iQ include
procedures fo.!.-Q.Q!ainir.:!Z lo~f f~~ack from the pr():l::u~t's users. Therefore, we developed
several questionnaIres and interview guides to use in the preliminary field test, and we trained
laboratory staff members to administer them.
Another important principle that applies to most educational R&D projects is to strive from the outset to
develop products that are fully ready for use in the schools. Partially developed products force local
practitioners to make adjustments to fill in the gaps. Because few schools are equipped to make such
adjustments, a partially developed product cannot be used effectively and is often badly misused. In the small-
scale R&D project described later in the chapter, the focus of development was a textbook chapter. This
product was not sufficient for field testing, however. The developer also needed to write a teachers manual so
that the field-test teachers would know how to use the chapter in their classrooms.

Preliminary Field Test and Product Revision

The purpose of the preliminary field test is to ol2J:ain an initial gualitatiYe e~tiQn of the new
educational product. For the minicourse this evaluation was based primarily upon the feedback of
a small group of teachers who take the course and the observations of laboratory personnel wHo
coordmate thefield test. As a rul~htteachs:rs have been sufficient for the preliminary field test,
since the emphasis of this evaluation is upon qualitative appraisal of course content rather than
quantitative appraisal of course outcomes.
In all phases of the R&D cycle involving product evaluation; it is important to establish field sites
similar to those in which the product will be used when it is fully developed. If a different type of field site is
used, the developer faces the problem of generalizing findings obtained in one setting to another. For
example, Minicourse 1 was designed to be used by elementary school teachers during their regular school
day. Therefore, the preliminary field test was carried out with six teachers from two elementary schools.
Instead of this procedure, we might have invited the teachers and some of their students to our laboratory to
take the course, perhaps on a speeded-up basis. The major problem with this procedure is that we might have
obtained a very unrealistic i~sion of the course. Elements of the course that raise no problem in a laboratory
setting might create havoc when used in the 'schools, causing an adverse effect on the course outcOine6'.
r",,~~
Throughout the preliminary field test of Minicourse I, two field representatives from the
laboratory worked closely with the six teachers in order to obtain as much teacher feedback and
observational data'as possible. Each teacher was interviewed individually three times during the
field test. These interviews focused upon specific problems and course deficiencies as well as
suggestions for improvement. At the end of the course, each teacher completed a questionnaire
regarding the course and participated in a group discussion with laboratory personnel. In addition
to these formal contacts, each teacher had informal contacts with one of the laboratory
representatives each day.
Teachers or other user groups receive a great deal of attention from developers because of
the need to receive extensive feedback from them. This attention can produce the Hawthorne
9
Effect, which will lead developers to overestimate the effectiveness of their product. Thus,
 developers must strive for a delicate balance in which feedback is obtained without giving the
field-test participants an undue amount of attention.
Observing the participating teachers near the end of the preliminary field test of Minicourse 1 revealed that the
teachers were generally unable to use the course skills effectively either in their regular classrooms or in their
microteach
lessons. Thus, from the standpoint of bringing about specific changes in the classroom behavior of
these teachers, the preliminary form of the course was a failure. End-of-course interviews and
questionnaires obtained from these teachers, however, indicated that they perceived the course as
being very effective and as providing them with a great deal of help in improving their teaching.
This experience suggests that global ratings are of little value in evaluating specific
educational objectives. Furthermore, they can be detrimental if they mislead developers into
believing that a product achieves its objectives and is ready for use when actually it is not. In the
case of Minicourse 1, the favorable testimonials were perhaps in part a result of the extremely poor
quality of most previous inservice teacher education programs, which teachers used as 'If standard
of comparison.
After the preliminary field test of Minicourse 1, all data were compiled and
analyzed. The development team used these results to replan the course and
then went on to make the necessary revisions.

Main Field Test and Product Revision

ain field test in th R&D c de is to determine whe er the educational
product U.Qder development meets its per ormance ~ves.Generally an experimental design is
used to answer this question. In the case of Minicourse I, a single-group pre-post design (see
chapter 15) was used to determine whether teachers would significantly increase their use of
qU$.stioning skills. About 50 teachers participated in the experiment. Shortly before the
course began, each teacher was asked to conduct a 20-minute discussion in his or her regular
classroom, and this discussion was videotaped. After the course was completed, each teacher
again conducted a 20-minute videotaped discussion.
Each videotape was viewed by trained raters who made quantitative observations of
teachers' use of the skills and behavior patterns presented in the minicourse. As each
videotape was coded and given to raters in random order, the raters did not know which were
pretapes and which were posttapes. Table
18.1 presents the major findings of the experiment.
Most of the changes in teacher and student behavior brought about by Minicourse 1 are
not only statistically significant, but are also Significant in their implications for educational
practice. Although a control group was not used in the main field test, subsequent studies
have indicated that teachers who take the course make substantially larger gains than teachers
who either do not have the course or who receive some form of minimal treatment.
In addition to the primary purpose of the main field test, which is to determine the success of the new product
in meeting its objectives, the secondary purpose is to collect information that can be used to improve the

TABLE 18.1
Main Field-Test Results from Minicourse 1

Posttape Mean Significance Behavior Compared (N 48) Level

Increase considered desirable
1. Number of times teacher used redirection. 26.69 40.92 4.98 .001
2. Number of times teacher used prompting. 4.10 7.17 3.28 .001
3. Number of times teacher used further clarification. 4.17 6.73 3.01 .005
4. Number of times teacher used refocusing. 0.10 0.02 0.00 NS·
5. Length of pupil responses in words (based on 5-min. samples of pre-and post-tapes). 5.63 11.78 5.91 .001
6. Length of teacher's pause after question (based on 5-min. sample of pre-and posttapes). 1.93 2.32 1.90 .05
7. Proportion of total questions that call for higher cognitive pupil responses. 37.30 52.00 2.94 .005

Decrease considered desirable

1. Number of times teacher repeated own questions. 13.68 4.68 7.26 .001
2. Number of times teacher repeated pupil answers. 30.68 4.36 11.47 .001
3. Number of times teacher answered own questions. 4.62 0.72 6.88 .001
4. Number of one-word pupil responses (based on 5-min. samples of pre-and post-tapes). 5.82 2.57 3.61 b .001
5. Frequency of punitive teacher reactions to incorrect pupil answers. 0.12 0.10 0.00 NS
6. Proportion of discussion time taken by teacher talk. 51.64 27.75 8.95 .001

course in its next revision. Therefore, questionnaire and interview data should be obtained
from all participants in the main field test.
If the main field-test findings indicate that the new product falls substantially short of
meeting its objectives, it is necessary to revise the product and conduct another main field
test. This cycle of field testing and revision would continue until the product meets the
minimum performance objectives set for it. In practice the product would probably be
abandoned if substantial progress was not made in the second main field test.

Operational Field Test and Final Product Revision

The purpose of the operational field test is to determine ~et~an educati2!:al p~t is fully ready
for use in the schools without the presence of the developer or his staff. In order to be fully
ready for operational use, the package must be complete and thoroughly tested in every
respect. In the case of the minicourse, all materials needed to coordinate the course are
normally tried out during the preliminary and main field tests. Because these field tests are
conducted by laboratory personnel, however, a satisfactory test of how well the total course
package works lion its own" cannot be made.
The operational field test is set up and coordinated by regular school personnel and
should closely approximate regular operational use. Feedback from both the coordinators and
the teachers taking the course are collected by means of questionnaires which are mailed to
the laboratory. The main use of these data is to determine whether the course package is
complete. Interviewers focus on parts of the course that fail to do their job or on materials that
are needed in order to make the operation of the course easier or more effective. Precourse
and postcourse videotapes are not obtained during the operational field test.
After the operational field test is complete and the data have been analyzed, a final
revision of the total course package is carried out. In the case of the minicourses, the Far West
Laboratory made a final revision of all scripts and printed materials and turned them over to a
commercial publisher for final production. The courses were then sold or rented to schools for
operational use in their inservice training programs.

Dissemination, Implementation, and Institutionalization

The R&D cycle is a time-consuming and expensive process. The way to justify the costs is by demonstrating
effective dissemination of the resulting product to its intended audience. Dissemination refers to the process of
helping potential users become aware of R&D products. Also, it is necessary to demonstrate that

the R&D product is implemented according to the developers' specifications so that it produces the intended
effects. Implementation refers to the process of helping adopters of an R&D product to use it in the manner
intended by the developers.
Successful implementation does not mean necessarily that the product wili be used on a regular,
continuing basis by the adopters. Therefore, developers also need to be concerned with institutionalization,
which is the process of making the R&D product an integral part of the adopting institution's structure and
 functions. The term "implementation" is sometimes used to include the institutionalization process. We
follow this convention here.
Despite the importance of R&D dissemination and implementation, these ( processes were seldom
studied until the mid-1970s. The concern of educational (R&D personnel prior to this time was on the
conceptualization and development of large-scale curriculum products using the R&D cycle of develop-
testrevise. Little funding was available for monitoring these products after they had been developed. Priorities
shifted dramatically in the mid-1970s, though. Many educators stopped using the term "research and
development," preferring instead to talk about "research, development, and dissemination" (R, 0, & D).
Research, development, and dissemination refers to the research-based development of products that meet
behaviorally defined objectives and dissemination and implementation criteria.
The ratio of 1: 10: 10 is sometimes used in industry to estimate funding requirements for R, 0, & D.
For example, suppose it requires $1 million to do the basic research for a new product. It will then require
$10 million to develop the product through the operational field test revision. Ten times that amount ($100
million) will be required to manufacture and disseminate the product.
Educators are not accustomed to think about the large sums of money implied by the 1: 10: 10 ratio for the
dissemination of R&D products. Commercial educational publishers do expend large sums of money for production
facilities, inventory storage and shipping departments, branch offices, advertising, sales forces, and inservice trainers.
Even today, though, these facilities and personnel are largely nonexistent in the federal and state educational systems.
For example, when the first minicourses completed their development cycle in the early 1970s, there were no official
plans either at the Far West Laboratory or at the U.S. Office of Education for their dissemination. A dissemination plan
was developed piecemeal with a commercial publisher. This plan was based largely on the publisher's established
distribution procedures rather than on a rational analysis of the dissemination and implementation requirements for the
particular product.
A dissemination and implementation capability for R&D products is slowly developing in this country. For
example, the National Diffusion Network (NON) was established by the U.S. Office of Education to
disseminate
1o
successful R&D products. This dissemination agency links successful products with school
systems that might benefit from them.
Dissemination and implementation logically occur at the end of the R&D cycle. This does
not mean, however, that developers can avoid thinking about these matters until then. Rather,
they need to consider dissemination and implementation issues in the initial stages of planning.
There is no point in developing a product whose target audience is difficult to reach or whose
implementation requires inordinately expensive staff development. By planning for
dissemination and implementation at the outset of product planning, developers may avoid
creating a white elephant.
If you are interested in educational R&D, one of your options is to do a study on some
aspect of the R&D process. For example, research on implementation processes has attracted
much interest in recent years. This research concerns such problems as the identification of
stages in the implementation process,l1 factors that facilitate or inhibit implementation,12 and
13
the use of staff development as a strategy for promoting implementation.

1...,/
PROBLEMS AND ISSUES IN EDUCATIONAL R&D

Virtually no R&D technology was available when the Far West Laboratory started its
development work in 1966. Therefore, we needed to develop our own procedures for handling
problems and issues as we encountered them. Because you are likely to face similar problems
and issues if you do an R&D project, we discuss some of them below.

L~arning versus Polish

The first problem concerns how far the educational developer should go in building the preliminary form of a
product involving an expensive component, such as instructional films or tapes. The development of the preliminary
form of Minicourse 1 presented us with an interesting dilemma. On the one hand, it was desirable to spend as little
money as possible on initial development, since the feedback obtained from the preliminary field test would almost
surely call for
extensive revision. On the other hand, a poorly developed set of materials might produce poor
results even though the ideas underlying the development were sound.
The most defensible resolution of this dilemma is to put most of the initial development
effort into a simple product that makes maximum use of learning principles, that is, a
theoretically sound product. Little or no effort should be devoted to such activities as
correcting minor errors in narration, building attractive charts where crude ones would serve
the purpose, or reshooting motion picture footage because of poor camera work.
In summary, our strategy called for giving the essentials our best effort and doing
everything else as cheaply and quickly as possible. This strategy was supported by the many
research studies that have found little effect on learning outcomes as the result of variations in
14
technical quality of audiovisual media.
Our experience indicates that developers will not find this an easy road to follow. Media
specialists on the development team, such as artists, actors, and television production
personnel, may apply great pressure to improve the nonessential aspects of the product. If
these pressures are not controlled by the developers, they will see most of their resources going
into unnecessary polish.
It is even more important to resist efforts to apply polish during the d~velopment cycle if
the product fails to achieve its objectives in field tests. One reason is that the developers have
spent a great deal of money, which they do not want to admit has been wasted. Second, the
product looks good, and some educators may use it for this reason even if it is not effective.
Finally, even though the product fails to achieve its objectives, it is easy to rationalize that it is
probably better, or surely no worse, than competing products currently in use.

Realism versus Pertinence

Another question of development strategy concerns the extent to which developers should strive to use typical
real-life examples in their products. Our work with model lessons for Minicourse 1 brought us face to face
with this question. Initially, we believed that our model lessons should be as realistic as possible. Therefore,
we started by selecting teachers who were reported to have outstanding teaching skills by principals and
supervisory personnel. We then worked indiVidually with each teacher, describing the skills that were to be
displayed in the model lesson and discussing in general terms methods for fitting these skills into a lesson and
modeling them effectively. We then brought videotape recording equipment into the classroom, and teachers
conducted the lessons that they had planned.
The typical outcome of these early efforts was a very long model lesson that contained
very few examples of the skills that we wished the teacher to model. For example, one of our
first model lessons ran for a full hour. During this time the specific skills that the teacher was
to model were demonstrated less than five minutes. Though providing a realistic picture of
typical classroom teaching, this model was extremely inefficient in terms of the objectives we
had set up for the model lesson. Furthermore, if this realistic lesson had been edited to reduce
the amount of time that the viewer was required to watch irrelevant behavior, the lesson
would have become highly unrealistic, since large segments would have been removed.
It became increasingly apparent that if the model lesson were to provide numerous
examples of the skills to be learned and contain a minimum of nonpertinent teaching
behavior, it would be necessary to plan the model lesson very thoroughly with the model
teacher. Thus, although the model lessons for Minicourse 1 were not scripted (the teacher and
pupils went through the lesson using their own words), they presented a less natural situation
than one finds in the typical classroom. In developing other minicourses, we found it
 necessary on occasion to prepare complete scripts so that the model lessons would provide
enough clear-cut examples of the skills to be learned within a reasonable period of time.
If you do an R&D project, you probably will find that examples are generally effective
in helping students learn the concepts, principles, and skills intended by the product. You
should not assume, however, that an example is effective just because it presents a typical
situation or is drawn directly from real life. Instead, you should test different types of
examples to determine which one's best help the learner achieve the product's objectives.

Other Lessons
Our experience in developing Minicourse 1 also taught us other lessons about educational
R&D. First, we learned that the rule so often stated by researchers-if anything can go wrong
in a research project, it will-seems to be equally true of R&D. For example, during the
preliminary development of Minicourse 1 in 1966, portable videotape equipment was still at a
rather primitive stage of development. Yet we were building a product that relied heavily
upon the use of this equipment for presenting instructional and model lessons and for
providing feedback during the microteach and reteach sessions. Therefore, we needed to
develop procedures that would reduce the problems caused by the bulkiness of the equipment
and its occasional malfunctions.
Finally~~:nJ.Q~~ee_~hat~ey..eillp~educational product was a far
more difficult and time-consuming task than we had anticipated. Major devel
opmentWorflii. educatIon-requires~alargeand-competentprofess:iunal staff and

significant long-term financial support. Surprisingly, we frequently encounter local school

administrators who wish to develop their own minicourses and other products. Very few school
districts have the resources to develop a validated product using R&D methodology.
Other fields sometimes solve the problem of obtaining the substantial resources needed for
R&D by forming collaborative arrangements. Projects that involve the sharing of personnel and
1S
facilities in universities and industrial firms are increasingly common. These projects require
resources that are unavailable to either organization working alone.
Similar to university-industry collaboration, some schools and universities have become
involved in collaborative arrangements for the purpose of doing R & D.16 Also, some of the
regional laboratories funded by the U.S. Office of Education are involved in collaborative R&D
projects. If you wish to do an
!~& D study for a thesis or dissertation, you should consider linking up with an
ongoing project of this type. These projects may be able to make resources
1\.1. vailable to you that would greatly facilitate your study. In return, your study
, ight contribute to the progress of the overall project.

AN EXAMPLE OF SMALL-SCALE R&D

We have already discussed the considerable resources required to carry out even a single
educational R&D project. It is highly unlikely that a graduate student will be able to find the
financial and personnel support to complete a major R&D project. In fact, educational R&D is
beyond the abilities of most school districts.
If you plan to do an R&D project for a thesis or dissertation, you should keep these cautions
in mind. It is best to undertake a small-scale project that involves a limited amount of original
instructional design. Also, unless you have substantial financial resources, you will need to avoid
expensive instructional media such as film and synchronized slidetape. Another way to scale down
the project is to limit development to just a few steps of the R&D cycle.
An example of an R&D dissertation is the project undertaken by Lawrence Cunningham to develop a history
textbook about the ancient Chamorros of Guam and an accompanying teachers guidebook.17 (The Chamorros are
the indigenous people of Guam.) The study of Guam history is a required subject in the Guam public schools.
Cunningham's long-term goal was to develop a complete textbook and guide, but he limited the scope of his
dissertation study
to one chapter of the textbook and the section of the teacher's guide pertaining to it. (Hereafter, the term
"chapter" refers to both the chapter and the teacher's guide.)
The objectives of the study were as follows:

1. to review the relevant literature on textbook instructional design and Chamorro history
2. to plan chapter objectives
3. to develop a preliminary form of the chapter
4. to field-test the preliminary form of the chapter
5. to revise the preliminary form of the chapter based on the field-test results
Is
6. to conduct a main field test of the revised chapter.

Each step of the R&D process used to develop the product is described in a separate chapter of the
dissertation. Chapter 2 presents the results of Cunningham's research and information-collecting activities.
These activities included a search for existing relevant curriculum materials, a study of learner characteristics
on Guam, a review of the literature on characteristics of effective text, and a review of the literature on the
ethnohistory of Guam.
Chapter 3 describes his initial planning activities, which focused on identifying objectives for the
proposed chapter. The following are examples of the objectives Cunningham identified:

1. Given pictures of 15 ancient Chamorro artifacts, you will be able to match at least 12 with their descriptive
labels.
2. Given paper, pencil, and a simulated situation in which you find an ancient Chamorro artifact, you will
state the proper things to do and not to do, as stated in this chapter.
3. Given a drawing of a latte stone, you will label the two parts of the latte stone and identify the latte stone's
purpose with 100 percent accuracy.
4. Given a map showing different environmental zones, you will identify good locations for building a
village. Your reply will be judged on the basis of what archaeologists have discovered about ancient
Chamorro settlement pat

terns.19

These objectives are written in the form of behavioral objectives, which were discussed earlier in the chapter.
Chapters 4 and 5 of the dissertation describe the development of the preliminary form of the product and the
preliminary field test. Two versions of the chapter were developed: an expository version (conventional text
format)
and a narrative version, which covered the same content but in a story format. Cunningham developed the two
versions because he was uncertain about which format would be more effective. It was feasible to develop and test
both versions because he limited the scope of R & n to just one chapter of the proposed textbook. In addition,
Cunningham developed a variety of evaluation instruments: a domain-referenced achie~ent test.£., a teacher
guestionnaire, student attitude scales, and a student irltervie; sdl;dule.. -.~----.--
~'--AtotaIof 16 sttidents'dra;~f;~~tw~-~epresentative Guam history classes participated in the
preliminary field test. They were formed into two groups, with one group studying the e~version and the other
group studying the narrative version. Both gr6ups completed each of the student instruments. In addition, the
materials were!~yj~~t=<:lJQ!,ClC:~lJXa<;:yRYJv.vQ!:m:J::,,,,'t~..2.1Q~ts; for community
accepta611itY~6yseveral.~h(:lJnQ1'!'()J.~~clgr§;..for quality of instructional design byan'Insfmcti§nal
tealnol~gist; for curriculum appr6E~i.~§sby one' of Guam's' associate school superint~l1dents;-afia-
fOrW!~SKgIsex..bias-=by eXE~i!s'on'tfiis-subject. The results of the field test were reported in the dissertatlon;-
and also the revisions made in the materials on the basis of the results.
 Chapter 6 of the dissertation presents the results of the main field test of the two revised versions of the
product. This field test involved a pretestposUest control-group experiment. The sample included five teachers and
four Guam history classes taught by each teacher (total N = 20 classes). Each teacher's four classes were randomly
assigned to the two treatment conditions-studying the expository version, or studying the narrative version. Each
teacher taught both treatment conditions, thus controlling for the variable of teaching effectiveness. Checks for
fidelity of treatment implementation were made.
Analysis of the experimental data revealed that both treatment groups made significant gains in
achievement and attitudes. No significant differences between groups on measures of these variables were
found. Table 18.2 presents a typical statistical analysis, in this case for the pre and post administrations of the
domain-referenced achievement test. In a subsequent face-to-face comparison of the two versions, the majority
of the students preferred the narrative version. Conversely, the majority of the teachers preferred the expository
version. Advocates of the narrative version felt that the story made ancient Guam history more interesting and
easier to learn. Advocates of the expository version felt that the story was a distraction from what they felt was
their primary task, which was to learn the information and pass a test on it.
Cunningham reached the following conclusions from these results:

It appears,then, that expository and narrative text work equally well in ecologically-valid situations. They both
contribute to student achievement. Therefore, decisions to choose expository or narrative text may be based on
other considerations, such as student preferences or the nature of the subject matter. 20
TABLE 18.2

Performance of Expository and Pretest M Posttest M Adjusted

Narrative Text Groups on (SO) (SO) Posttest M
Achievement Test in Main Field
Test Treatment Group N F
Expository text 157 5.25 (3.91 ) 24.97 (7.88) 25.04 .00
Narrative text 156 5.55 (4.14) 25.12 (6.86) 25.05

If you are planning to do an R&D project, you should give careful A~( /~consideratiOn to the time
required. The dissertation described above took well ~'i / lover a year for completion of product development
through the field-test phase.
/1 I. A research project for the master's thesis or doctoral dissertation typically can be / [completed in much
less time. The additional time required for an R&D project
is worthwhile, however, if you are interested in making a contribution that will
lead to an immediate tangible improvement in educational practice.
In the case of the dissertation described above, the developer was able to make a contribution not only to
practice but also to research knowledge. You will recall that his main field test involved an experimental
comparison of expository and narrative presentation of information. The results of the field test contributed
new knowledge, and raised new questions, about the effects of variations in text characteristics on learners. In
planning an R&D project, you \ \ too may find yourself considering alternatives about suchl!l~a.uJroduct ; design,
product conterr.t. and target audience. It may be possible to compare several alternatives through informal or
systematic experiments incorporated in the field-test phases of the R&D cycle.
The dissertation described above took the development of a product through the main field-test step of the
R&D cycle. It may be feasible to take other products through all steps of the cycle. For other products, the
development task may be sufficiently complex to justify a dissertation study that ends at the preliminary field-
test phase. You and your dissertation committee will need to consider the nature of the proposed product and
decide how much of the R&D cycle would constitute an acceptable study.

MISTAKES SOMETIMES MADE IN DOING

RESEARCH AND DEVELOPMENT
1. The developer does not draw upon research-based principles of instructional design in planning a product.
2. Does not determine at the outset whether there is a need for the product.
3. Overlooks certain aspects of the product in listing topics to be included in the literature review.
4. Does not state the product's objectives in a form that enables them to be measured clearly.
5. Obtains insufficient feedback from users during each field test of the product.
6. Does not plan for the dissemination and implementation of the product while it is still under development.
7. Pays too much attention to the cosmetic aspects of the product and too little attention to its instructional
effectiveness.