1

Critique of a research report – EDUC321

 2

Introduction and purpose

Research is an act that lies at the heart of scientific inquiry. It revolves around searching,

identifying and analyzing relationships, understanding them and producing valid knowledge for

the community to benefit from. However, for valid knowledge to be produced, the researcher has

a roadmap to follow intended primarily to control for all the biases that may interfere with the

objectiveness of the approach thus resulting in the distortion of the findings. In fact, Gall, Gall

and Borg (2014) argue that to produce sound and generalizable knowledge, special methods will

have to be used by the researcher to avoid undermining the soundness of the findings. These

special methods do not pertain only to the procedural aspect of the research, but equally to its

final presentation in the form of a research study. In fact, if other researchers, practitioners and

the community at large are to benefit from the generated knowledge then findings will have to be

presented in a manner that allows for an understanding of the research through a logical

development of ideas that observes the typical format of a research study consisting of the

following sections: Abstract, Introduction, Methods, Results and Discussion (Gall, Gall and

Borg, 2014). The purpose of the present report is to identify and critique the various aspects of

Lamb and Booker’s (2004) research that reports on the effect of teachers’ conceptual knowledge

of division on students’ knowledge of division in Math education. As stated earlier, this critique

will address all the aspects of the study both at the organizational and procedural one.

Organization of the report

Overall, this research observes the typical sections of a research report with few

exceptions however. The paragraph entitled THE STUDY integrates the purpose of the study and

its methodology. Similarly, the next paragraph equally integrates the results and their discussion.

Martin Stute, a professor of Environmental Science at the University of Columbia argues for
 3

instance against the grouping of independent sections as this may hinder the understanding of

ideas especially that the purpose of each section would be to inform the following section thus

allowing a smooth progression of ideas (Stute, University of Columbia). Gall et al. (2014)

equally emphasize the importance of a research report having the usual report sections. Lamb

and Booker’s (2004) study should abide by the typical format of a research report in terms of

organization of sections for a greater coherence and progression of ideas.

Research purpose and research questions

A well written research report should include plainly the purpose of the research.

According to Gall et al. (2014), the purpose statement’s importance resides in the fact that it

“gives direction to the design of the study” (p. 26). Lamb and Booker’s (2004) research seems

problematic on that aspect. The reader does not find the purpose clearly stated in the research,

even more, it seems stated in a nuanced manner in two places in the study. In fact, after a brief

review of literature that emphasizes the importance of a conceptual understanding of math

operations, the authors of the report conclude the introduction section with a generic statement

saying that it would be of interest to investigate the division understanding of teachers and their

students to have better insight about students’ difficulties with division. In the subsequent

section, the purpose of the study becomes more refined when the researchers indicate that they

are analyzing teachers’ conceptual understanding of division and its relationship to students’

knowledge of division. It is in fact the understanding at the conceptual level of division that

seems to drive the research and distinguishes the two phases of the study. In a similar vein, the

study should have at least included research questions to elicit its purpose and direction. Possible

research questions in this case could have been:

 4

- What is the impact of teachers’ understanding of division on their students’

understanding of division?

- To what extent does teachers’ changed understanding of division after professional

development impact students’ understanding?

The first part of the RESULTS AND DISCUSSION section clearly answers the first

proposed research question by diagnosing in phase 1 the current understanding of division of

teachers and their students whereas the second part of this section diagnoses in phase 2 how

introducing games and concrete materials by teachers has improved students’ understanding of

division.

Building on previous research and Rationale

The rationale of the report and the way it builds on previous research seems reasonably

founded. In fact, in the INTRODUCTION section Lamb and Booker (2004) survey briefly the

literature to justify how their study builds on previous research. They start first by highlighting

the importance of division in various life situations and note that some researchers argued that an

understanding of the thinking required by this algorithm differentiates between a secondary

school-level thinking and a primary level one. The authors introduce then the conceptual level of

thinking involved in division by reviewing some research findings about the topic to ask later if

the teaching of math should aim more towards a focus on conceptual or procedural

understanding. At this point, the researchers evoke one main study (Ma, 1999) that compares US

and Chinese teachers’ conceptual understanding and procedural ability on various math topics.

They survey next a set of researches that study several reasons associated with students’

difficulties in understanding division. However, according to them, no research attempted to

study a possible relationship between teachers’ conceptual understanding of a specific type of

 5

math topic; the division, and their students’ understanding of it. This seems a valid rationale that

builds on previous research and contributes to the generation of new knowledge. However, the

researchers could have still reviewed additional literature to provide a greater solid foundation

for their research.

Significance of the research

On the aspect of stating clearly the significance of the research in terms of its

contribution to Educational Research and Practice, this study presents weaknesses. There is no

clear reference in the entire article of the intended significance of this research and the reader has

to dig that out by assuming what this study’s significance may be. In fact, as stated above, what

the researchers claim here is to investigate a relationship between teachers’ conceptual

understanding of division and their students’ understanding of it. Consequently, it can be

assumed that from an educational research perspective, the researchers hope to raise other

researchers’ interest in investigating further this relationship and the impact of professional

development on improving teachers and students’ understanding of division. From an

educational practice perspective, this research attempts probably to advise practitioners such as

policy makers and school directors about the importance of professional development on

students’ class achievements. In all cases, it should not be left for the reader to assume the

significance of a research study rather, it falls on the researchers’ responsibility to indicate it

clearly.

From this point on, the research presents important methodological deficiencies that will

be discussed in what follows.

 6

Population, sampling and sample used in the research

To study the relationship between teachers’ understanding of division and their student’s

understanding of it, the researchers use a sample of two classes from “year 7” and their

corresponding teachers with one class (24 students) and their teacher used in phase 1 of the

experiment and another class (23 students) and their teacher in phase 2. From a population

perspective, the researchers are not explicit about the target population to which the research is

supposed to be generalized. Is this research valid only for “year 7” students in schools or is it

generalizable to all elementary school years? Another detail left unspecified is the type of school

(public or private) and their geographical location to which the research is addressed. This detail

may be important due sometimes to curricular differences. According to Gall et al. (2014), it is

important to establish population validity by proving that the variables studied related to the

research are similar among the sample, the accessible population and the target population. With

the absence of indication of the target and accessible population, population validity seems

difficult to establish and consequently the results’ generalizability compromised. From a sample

perspective, the type of sampling used in this research is cluster sampling where participants

with similar characteristics are naturally grouped. Here, the researchers should have also given

further indication about the criteria based on which the classes and their teachers were selected

as well as the sample demographics such as gender distribution for example. Voyer and Voyer

(2014) argued in their findings for instance for the presence of a stable female advantage in

school achievement. Additionally, information should have also been given about the age

difference of the two teachers as their different conceptual understanding of division may be due

to the fact that one of them is a fresh college graduate. Finally, six students were selected for the
 7

subsequent interviews and the researchers do not provide any detail about the selection criteria of

these students.

Research instrument

The research instrument represents another flaw in this study. In this case, the instrument

used is the pencil and paper test that contains the six division questions. Although pencil and

paper tests are widely administered in educational research, one of their limitations resides in the

difficulty of insuring that participants are in a sound physical and mental state (Gall et al., 2014).

However, this may not constitute the essential flaw of the research instrument, rather other

concerns including its validity and reliability are legitimately raised. First, there is no indication

of the conditions under which the test was administered and who supervised its administration.

Catering for such conditions insures that all participants had equal and fair chances in completing

the test. Second, there is also no indication if the test items are aligned with the division level of

“year 7” students. In this case, it would be difficult to determine if the instrument is adequate to

be used in the first place in the research. Additionally, although the researchers indicate that the

test is supposed to measure teachers’ and students’ understanding of division they provide no

evidence of its content validity. While providing the reader only with questions 5 and 6, it is

uncertain if the other questions of the instrument measure the intended objective and

consequently difficult to justify the overall test validity. In the absence of such information, the

researchers should have at least indicated if the content validity was established by expert

judgement. Additionally, there is no evidence of the reliability of the test. Given the lack of

information about the instrument, we cannot be certain if the instrument is consistent across

conditions, forms and testing situations. The absence of indication of Cronbach’s alpha internal

test’s consistency casts additional doubts about the quality of the instrument used in the research.
 8

Although the interviews are not part of the instrument used they help in giving further insight

about the data collected quantitatively. Here again, the researchers are silent about the capability

of interviewers in conducting interviews, the manner how the interview was recorded and if it

carries any bias with it and whether some questions may have made students feel uncomfortable.

Research design

To investigate the relationship between the independent variable represented by teachers’

conceptual understanding of division and the dependent variable represented by students’

knowledge of division, the researchers use a group comparison research design within the non-

experimental research paradigm. In fact, the researchers choose one class and their teacher in

“year 7” and compare the studied variables to another “year 7” class and another teacher one

year later. According to Gall et al. (2014), this is a model of research design that compares

groups on the dependent variable (students’ knowledge of division) to find if they also differ on

the independent variable. Gall et al. (2014) consider also that in critiquing a research study, it is

important to see if the researchers have defined their variables in a way that is consistent with

other definitions in the literature. Although the researchers in this study do not provide a clear

definition of their variables we may deduce that their understanding of the conceptual knowledge

of division revolves around using games and concrete representations of the concept of sharing

related to the division. Whereas the researchers remain silent about if this is also the definition

used in the literature they surveyed (Ma, 1999 for example) it is known that the division could

also be linked to the concept of grouping as well. Additionally, the way they used this design

represents an unusual way for the use of the group comparison research design that will feature

weaknesses and question the validity of the findings. In fact, a group comparison research design

is usually conducted between two groups at the same point in time. In this research, comparing
 9

the two groups a year apart does not allow the researchers to control for changes that may occur

during the experiment. For example, the “year 7” class used one year later could have had

curricular changes in math the previous year which may result in a better knowledge of division

regardless of the professional development that occurred. Similarly, the math teacher part of the

research the next year could have been newly appointed and her improved understanding of

division could be due to the fact the she’s a fresh college graduate. Consequently, the

comparison groups are not necessarily similar in all respects except for the variables and this will

eventually question the findings. A better group comparison design would have probably been to

choose among two “year 7” classes during the same year with one teacher who followed

professional development sessions and another teacher who did not and compare the dependent

variables. Or if all teachers in the school had followed professional development the researchers

could have chosen a “year 7” class from that school and another “year 7” class from another

school whom teachers were not given professional development sessions and compare between

these two classes. Obviously, in this second case the researchers will have to control for many

conditions such as making sure that the two schools follow the same math curriculum in year 7

classes. Another research design that could have also been used in this case is a quasi-

experimental research design where professional development is the experimental intervention.

In this case, the researchers choose a “year 7” class as an experimental group and another “year

7” class as control group.

Data Analysis used in the research

As stated earlier, the researchers should have devoted a paragraph to explain their

methods of data analysis and not integrate them with the RESULTS AND DISCUSSION

section. Their approach to data analysis was through comparing independently overall results
 10

between phase 1 and 2, error analysis of test responses and analyze the way in which remainders

are interpreted. In the first table that compares overall results, the researchers use percentages to

show an improvement in the understanding of division between phase 1 and 2. However, the

only use of percentages in the data analysis seems to be a somewhat weak approach to analyze

data. Additionally, the researchers do not explain how they computed the percentages provided

in the different tables. Conversely, they could have used descriptive statistics by calculating for

example the means and standard deviations on the dimension “correct answer with calculation

shown” indicated in their table in phase 1 and 2. In fact, comparing the mean scores on two

groups is a standard way of analyzing data in a group comparison design. A “t-test” could have

also been used to determine whether the two means in each phase differ significantly. In fact, the

researchers could have calculated the score of each student on the test in phase 1 and phase 2 and

perform a paired-sample t-test to observe whether there is a statistically significant difference

between the means that are due to professional development. There’s also the possibility of

conducting a one-way ANOVA test if the independent variable (teachers’ conceptual

understanding of division) was grouped for example in three levels (high-medium-low) and test

the hypothesis that the mean scores of students is equal in both phases. Similarly, such tests

could have also been conducted in the error analysis dimension and the interpretation of

remainders. Concerning the qualitative data that emerged from the interviews, the researchers are

completely silent about the way data was analyzed, if specific coding procedures were used and

if new themes that would add to the quantitative data emerged. Instead, they refer to a simple

display of information.
 11

Research Results

As said above, using percentages is a weak approach to analyzing data but the

percentages provided by the researchers indicate an important improvement of students’

understanding of division after professional development. The results from the interviews as

explained by the researchers confirm the results displayed by the percentages and show that in

phase 1, students performed division in a procedural manner that does not grant conceptual

understanding and they miss pronounced the division as well. They also had a high percentage of

process errors and a low priority of interpreting remainders. The teacher in phase 1 justified this

by a teaching method that emphasizes procedural knowledge. In phase 2 however, students

exhibited a greater conceptual understanding of division with less process errors and a better

interpretation of remainders. The teacher in this phase used concrete materials to raise students’

conceptual understanding of division however the interpretation of remainders still was not

prioritized by the teacher.

Research discussion

In their discussion, the researchers concluded that professional development has

improved the students’ conceptual understanding of division but not at the level of interpretation

of remainders. Obviously, the flaws in the research design explained above leave great room for

rival explanations of the findings and these were not accounted for by the researchers in this

section. The discussion is also poor on the conceptual implications of their finding in relation to

the theories they had reviewed in the literature (Ma, 1999 for example). Although their findings

assume that professional development has led to an improvement of students’ conceptual

understanding of division, this fact is not explicitly indicated in this section. Finally, there is no
 12

indication of the limitations inherent to this study and how further research can be conducted in

the future.

Conclusion

Based on all the previous discussion, and the importance of research as a means of

generating valid and generalizable knowledge as explained in the introduction of this critique,

the present research needs major improvements on the conceptual, design and methodological

levels and cannot be accepted as a valid research report to be discussed in a conference.

 13

References

Class material and notes

Gall, J. P., Gall, M. D., & Borg, W. R. (2014). Applying educational research: How to Read, Do,

and Use Research to solve problems of practice (6th ed.). Pearson Education

Lamb, J., & Booker, G. (2004). THE IMPACT OF DEVELOPING TEACHER CONCEPTUAL

KNOWLEDGE ON STUDENTS’KNOWLEDGE OF DIVISION. In Proceedings of the

28th Conference of the International (Vol. 3, pp. 177-184).

Stute, University of Columbia; retrieved from:

http://www.ldeo.columbia.edu/~martins/sen_sem/results_discussion.html

Voyer, D., & Voyer, S. D. (2014). Gender differences in scholastic achievement: A meta-

analysis. Psychological Bulletin, 140(4), 1174-1204. doi:10.1037/a0036620