Differentiating between Sources of Numerical and Non-Numerical

Data

Reliability and Validity

Numerical data are objective and quantifiable bits of information about a population from
methods like assessment, experimentation, observation, and survey. They are collected using
instruments like achievement tests, attendance registers, archived data, assessment rubrics,
and questionnaires, to name a few.
However, instruments must be so designed that they produce VALID and RELIABLE data.
The RELIABILITY of an instrument is its ability to consistently produce similar results with the
same group, under the same conditions.
The VALIDITY of an instrument is its ability to measure what it is intended to measure, for a
specific group of people. It has to do with how well interpretations from data align with the
purpose of the instrument.

Strengths of Checklists
 Record information quickly/immediately
 Directed to individuals or group
 Potential for more accurate data
 May identify problem areas of an individual student.

Weaknesses of Checklists
 Items are vague and open to interpretation.
 Identical scores may hide differences among individuals
 Reliance on memory after-the-fact may bias data.
 Does not account for the context of behaviour/skill
 Narrow focus might overlook other information

Reflection

 What type of observational method might you use in your action research?
 How will it help you collect data you need to answer your research questions?
 What descriptors might you include in your checklist?
 Methods of Collecting Numerical Data - Journals
A journal is “a sequential, dated chronicle of events and ideas, which includes the personal
responses and reflections of the writer (or writers) on those events and ideas” (Stephens &
Cooper, 2009).
Teachers’ journals provide insight into their analyses, interpretations, and reflections on practice
over time (Cochran-Smith & Lytle, in Mills, 2007).
Students’ journals provide insight into their changing thoughts, new ideas, and the progression
of learning (Anderson, Herr, and Nihlen, in Mills, 2007).

Strengths of Journals

 Facilitates critical thought

 Access to private thoughts not readily offered in interviews
 Instructional tool for teacher
 Learning tool for student
 Can develop problem-solving skills through reflection on thinking
 Develops writing and communication skills
 Can be an assessment tool

Weaknesses of Journals
 Highly subjective
 Participants attitudes to keeping them may vary
 Time-consuming
 Participants may lack the necessary writing skills for effective communication

Reflection

 How will journals help collect data to answer your research questions?
 What information would journals provide?
 Who will keep a journal? (researcher/participants)?
 Methods of Collecting Numerical Data - Achievement Tests and
Rubrics
Achievement Tests
Because you have covered achievement tests in Pedagogy as Process, there will be no detailed
treatment of them here. However, have a look at some of their strengths and weaknesses.

Strengths of Achievement Tests

 Focused on unit content.
 Same stimulus provided to all participants.
 Allows comparison among students

Weaknesses of Achievement Tests

 May be biased against certain sub-groups within a sample.
 Potential unexplained non-response to selected items.
 Identifies what learner recalls/can do at that time.
 May not produce valid/reliable data.
 May be affected by external factors like anxiety/fatigue.

Rubrics
Rubrics provide written expectations of a specific task, through multidimensional scoring criteria
that facilitate efficient, consistent, objective, and quick scoring.

Strengths of Rubrics
 Provide consistent scores across all students.
 Define expectations for performance.
 Provide focus, emphasis, and attention to particular details of performance.

Weaknesses Strengths of Rubrics

 Complex/time-consuming to develop, test, evaluate.
 Require continuous revision.
 May restrict students’ initiative.

Reflection

 Do my research questions suggest the need to use achievement tests to collect data about
participants?
 What information about participants will they provide?
 When and how will the test(s) be administered?
 Will I need rubrics for assessments I design?
 How will rubrics inform what I can say about my participants?
 Methods of Collecting Numerical Data
Surveys and Questionnaires

A survey is a method of “questioning individuals on a topic or topics and then describing their
responses” (Jackson, 2011, p.17). You might be interested in participants’ knowledge, attitudes,
perceptions, or behaviours, at one point in time or over time. Surveys are guided by the
research questions, and systematically collect data through questionnaires.
Questionnaires are instrument comprising various types of questions/items to gather
information from respondents. They collect information to describe, compare, explain
knowledge, attitudes, perceptions or behaviours.
Questions or items on a questionnaire must be valid, reliable, understandable, address a
single issue, devoid of jargon, have a meaningful scale (where appropriate). Questions may
also be open-ended.

Strengths of Questionnaires
 Easy to administer
 Quick to fill in
 Collects much data quickly
 Compares groups and individuals
 Provides feedback on attitudes, adequacy of resources, preparation for the next session,
conclusions at end of term ... and so on

Weaknesses of Questionnaires
 Analysis is time-consuming
 Items may not measure construct accurately/in-depth
 Relies on respondents’ reading ability and comprehension
 Responses may be inhibited
 Children, especially, may try to produce ‘right’ answers

Reflection

 How will it help you collect data needed to answer your research questions?
 What types of questions or items might you include in your questionnaire?
 Can I find one that has already been tested? Does it match my study? Do I need permission
to use it?
 Methods of Collecting Numerical Data - Final Words

During the Research sessions, you have been exposed to numerous methods for collecting
numerical and non-numerical data. It might be challenging for you to determine which methods
and instruments are appropriate for your research.

Here are two final questions that might help you determine what methods would be most
appropriate for your action research as you continue designing your study.

1. If data are to be collected from you, the teacher-researcher, what methods might help with
answering your research question?
2. If data are to be collected from students/teachers (for EdAdmin), which methods might be
most useful for accessing the data?

It might be helpful for you to use a triangulation matrix to map your research questions to
appropriate research methods and instruments. Feel free to use the table below to select the
best methods and instruments for your action research.