A scientific approach to educational research?

Whats this all about?

Understanding the assumptions behind the scientific approach to educational research Understanding the power and limitations of experimental designs and of statistical tools Understanding the potential of mixed methodologies and methods

Supporting critical reading and personal research decision making

Positivist Assumptions
after Lincoln & Guba 1985
There is a single reality that can be revealed; this reality will not be contested by fair minded individuals Part of this can be studied independently; the whole is the sum of those parts It is possible to study this objectively, independently of the researcher. Results will apply at other times; in other places Cause and effect can be distinguished Inquiry is value-free

True of science?
Science is about building models not mapping reality. Different models are possible from the same data Data has error so no picture of reality is error free (triangulate across forms of data, alternative studies etc) 0+0 = 100 Observing a system necessarily changes it x. ph Sometimes causes are complex and not open to definition Values determine what science is done and what is regarded as important

True of social science?

If its not really true of physical science, its not likely to apply cleanly to social science BUT ALSO Scientific social research is dependent on the logic of statistical significance
Meaning of statistical significance Limitations of statistical significance

Issues of sampling

Dependence on the logic of statistical significance

An example
Group 1 10 people, average score=6

Group 2 10 people, average score=8

Is this difference between the groups important? FIRST STEP: is it likely to occur just by chance?

Basic notion of significance

Assume no difference in outcome between the two groups. This is our NULL HYPOTHESIS Ho The groups are like random samples from the same population. Would it be unusual for two random samples from one population to have means as different as 6 and 8?

If yes, we could assume that the two groups came from different populations
Difference is statistically significant. Reject Ho

If no, we would say that the difference is probably just down to chance
Difference is not statistically significant. Accept Ho

Population
8 5 10 4 7 6 4 4 0 5 3 8 6 2 3 1 9 4 0 9 9 7 3 5 5 4 7 3 7 6 9 10 0 7 4 9 2 7 7 7 2 5 4 7 4 8 3 1 8 7 1 5 9 1 1 7 6 2 7 1 7 7 0 9 3 5 6 9 6 2 5 6 4 3 7 7 3 3 9 10

One random sample

8 5 10 4 7 6 4 4 0 5 3 8 6 2 3 1 9 4 0 9 9 7 3 5 5 4 7 3 7 6 9 10 0 7 4 9 2 7 7 7 2 5 4 7 4 8 3 1 8 7 1 5 9 1 1 7 6 2 7 1 7 7 0 9 3 5 6 9 6 2 5 6 4 3 7 7 3 3 9 10

Mean of red group = 7.1

Two random groups

8 5 10 4 7 6 4 4 0 5 3 8 6 2 3 1 9 4 0 9 9 7 3 5 5 4 7 3 7 6 9 10 0 7 4 9 2 7 7 7 2 5 4 7 4 8 3 1 8 7 1 5 9 1 1 7 6 2 7 1 7 7 0 9 3 5 6 9 6 2 5 6 4 3 7 7 3 3 9 10

Mean of red group = 7.1; mean of blue group = 5.9 Difference = 1.2

Distribution difference of two means

Distribution difference of two means

Not just group differences

Is there a correlation between two variables? One group 10 people IQ and attitude scores are linked

Correlation
IQ and attitude to school
180 160

140

120 100

80

IQ score

60 40 0 10 20

attitude to school

Correlation
Correlations attitude to IQ s core s chool 1.000 .564** . .000 40 40 .564** 1.000 .000 . 40 40
IQ and attitude to school
180 160

IQ s core

attitude to s chool

Pears on Correlation Sig. (2-tailed) N Pears on Correlation Sig. (2-tailed) N

**. Correlation is significant at the 0.01 level (2-tailed).

140

120 100

80

IQ score

60 40 0 10 20

attitude to school

Statistically significant?
Is it unlikely that we would get this correlation if we just chose 40 pairs of numbers from a set of random numbers? If it IS unlikely, then our results probably arent just pairs of random numbers there is a link between IQ and attitude If is NOT unlikely, then our result could well be nothing more than chance

The importance of significance

It may be significant but does it matter?

The importance of significance

It may be significant but does it matter?

Effect of sample size

Critical Values of r for Rejecting the Null Hypothesis (r = 0) at the .05 Level Given Sample Size n
n 3 5 10 20 50 100 500 1,000 5,000 10,000 r .997 .878 .632 .444 .276 .196 .088 .062 .0278 .0196

Note: Values are taken from Table 13 in Pearson and Hartley (1962).

What if? arguments

If we get a significant result with a sample size if 100 how small would the sample have to be for this to become non-significant?
(If very small fairly confident in rejecting Ho; if 90 (say) not very confident in rejecting Ho)

If we get a non-significant result with a sample size if 20 how big would the sample have to be for this to become significant?
(If 30, say, not much confidence in Ho; if very large fairly confident in Ho)

Effect size

Effect size m1 m2 sd1

Effect Size =1 - the mean for one group coincides with boundary score for the top 16% of the other group

Non-significant results

Sampling
Samples that allow statistical generalisation
random systematic stratified random cluster multi-stage

Samples that dont allow statistical generalisation

quota convenience snowball

Sampling
Samples that allow statistical generalisation
random systematic stratified random cluster multi-stage

Samples that dont allow statistical generalisation

quota convenience snowball

Is your sample representative?

Since its quite difficult to get representative samples, the answer may well be no.

The experiment
SMs gold standard for revealing cause and effect relationships

The experiment
Logic of experimental methodology Avoiding threats to validity Problems
Sampling and assignment Models of causation

Solutions
Quasi-experimental designs Mixed methods within an interpretive paradigm

OXO
Is X the cause of any difference between the first and second observation?

Issues of validity at the design stage experimental designs

Internal threats to validity History Maturation Testing Instrumentation Selection Statistical regression Mortality

External threats to validity Interaction of selection bias and treatment Interaction between testing and treatment Reaction to being in an experiment

True experiment

X -

Oe
Oc

Why are experimental designs so useful?

X Oe Oc

They deal with treats to validity. ..??

So why bother with SM at all?

Can be very powerful in revealing complex interactions and relationships

Multiple regression
Dependent variable English O-level Independent variables
DAT scores (VR,NA,VN,SR,MR,CSA,SP,LU) Attitudes ( A, B, .I) Creativity (Flu,Flex,Orig)

Eng O level = 0.65LU + 0.35SP +0.66VR +2.64B +31.26

Adjusted R squared = 0.36 Standard error = 15.18

Chooses
Stays on task
85 56

Persists at task Eager to learn

Infant checklist

Match pictures
43
48

Draws onjects Can use tools

59 49

83

Copy cross Copy square

58

Picks up songs Describes env Organised tale Listens to story Gives message Clear speech

59

42

70

72

58

40

43

Settles away

61

Receptive to sch

79

Integrates-peers

60

Plays - peers

64

Relates - teacher

72

40

Cluster results

So why bother with SM at all?

Can be very powerful in revealing complex interactions and relationships So how do we get the benefits in spite of the problems?
through limiting the population through non-inferential stats through links with interpretive designs

Limiting the population

Non-inferential use of statistics

Quasi experiments no R

O O

O O

Using quantitative methods within an interpretive methodology

Mixing approaches?
Paradigms as watertight boxes simply a choice to be made Paradigms as coherent systems that serve as the starting point for creative thinking

One example
Using interpretive methodologies to explore possible threats to validity in a quasi-experimental design

Interpretive Assumptions
after Lincoln & Guba 1985

Realities are multiple, and are individually constructed The knower cannot be separated from the known We can only make statements that are time and context bound All entities are continually shaping each other Inquiry is inevitably value-bound

Creative thinking
Linking qualitative and quantitative data qualitative work gives rich exemplification of generalisable relationships established by statistical methods (Sci Paradigm)

quantitative work establishes the generalisablity of hypotheses which emerge from a qualitative enquiry (Sci Paradigm) qualitative and quantitative work are used together (iteratively) to deepen the understanding of the particular cases on which we have been working. (Interp. Paradigm)

Its NOT the purpose of qualitative work simply to give rich exemplification of generalisable relationships established by statistical methods to give a human face to a statistical study. NOT that quantitative work should be used to establish the generalisablity of hypotheses which emerge from a qualitative enquiry - as if this is in some way a necessary step in order that the qualitative findings can be taken seriously. BUT qualitative and quantitative work are used together (iteratively) to deepen the understanding of the particular cases on which we have been working.

Another example
Using scientific and interpretive methodologies iteratively to explore factors influencing learning in FE classrooms

C1 site differences
3.5 SpprtMS WrkbdAst 3.0 ITSkills Connect2

Shared Control > Student Negotiation

ELDrama ESOL LS

Shared Control

2.5

Pth4 Prn

GNVQBus

ADMPA

Student Negotiation > Shared Control

2.0 AVCET&T Voc Path BTECHlth Engneer 1.5 2.0 2.2 2.4 2.6 2.8 3.0 3.2 3.4 ASPsych CACHE

Student Negotiation

Explaining the High SC/Low SN grouping

Support for Mature Students Self assessment, negotiation based on assignments, individual learning plans agreed and reviewed by tutor and student Workbased assessment Individual support from tutor (underground working)

Explaining the High SC/Low SN grouping

Workbased assessment Different geographical placements

Support for Mature Students and IT skills

Same room
but ISOLATION

Different times ESOL Same room, same times

but

Different languages

Does isolation feature elsewhere?

3.5 SpprtMS WrkbdAst 3.0 ITSkills Connect2

Shared Control > Student Negotiation

ELDrama ESOL LS

Shared Control

2.5

Pth4 Prn

GNVQBus

ADMPA

Student Negotiation > Shared Control

2.0 AVCET&T Voc Path BTECHlth Engneer 1.5 2.0 2.2 2.4 2.6 2.8 3.0 3.2 3.4 ASPsych CACHE

Isolation was a feature of the top left sites In 4 of the bottom right sites isolation was not at all evident in this site

Student Negotiation

Isolation (broadly defined) appeared to be a factor related to a site culture in which there was low student negotiation.