Beruflich Dokumente
Kultur Dokumente
165
From that work, the peer discussion portion of the PI process was Table 1: Dataset Characteristics including Week 1 Enrollment
shown to improve student understanding [7]. Characteristics.
166
Table 2: Student course outcomes among CS/E majors and non- “kept” in the pipeline and students we “added” to the pipeline
majors including (M)ales, (F)emales, and (O)verall. (e.g., changed into our major).
CS1-Before CS1-BP Statistical significance. We determine statistical significance
Previous Version New Version using a two-tailed two-proportion z-test for two independent
groups. These groups were created by summing students across
CS/E Majors Drop by classes to produce a group of students who had taken CS1-Before
10%/ 15%/ 11% 1%/ 2%/ 2%
Week 4 deadline (M/F/O) and another group having taken CS1-BP.
CS/E Non-Majors Drop by
Week 4 deadline (M/F/O)
17%/ 22%/ 19% 17%/16%/16% 3.1 Threats to Validity
As a retrospective study of real courses, there are a number of
CS/E Majors Passing 677 587 threats to the validity of this study.
CS/E Non-Majors Passing 663 424 Instructor Variation — Instructors may vary in their appeal to
Pass rate Majors (M/F/O) 80%/ 81%/ 80% 93%/96%/94% students and, potentially, their ability to retain students in the
major. To isolate this issue, we present a within-instructor
Pass rate Non-Majors analysis, looking at the results of just that instructor who taught
73%/ 72%/ 73% 86%/84%/86%
(M/F/O) both versions of the course four times (for a total of eight). This
Gender of Passing Non- assumes that the instructor was equally charismatic in all classes
69%/ 32% 61%/ 39% (which cannot be measured).
majors (M/F)
Gender of Passing CS/E Change in Students — Student abilities, perceptions of
81%/ 19% 76%/ 24% computing, and interest in the major may have shifted during the
Majors (M/F)
study. Specifically, students enrolled during our standard course
(2001-2008) may differ from our new course (2008-2011). In the
the impact of CS1—before other factors (later classes, instructors, following section, we examine a different advanced-track CS1
etc.) add additional noise. For the purposes of this study, we course not part of this study for changes in these two time periods.
define the following forms of retention (note all denominators are
the same): Impacted vs. not Impacted — The CS/E major was impacted
(had higher standards for admission) for much of the time period
Programmatic Retention: (#of CS1 students taking a of the standard course (2001-2007) and was not impacted during
computing course and majoring in CS/E one year later) / the duration of our new course (2008-2011). Again, we compare
(#of students in CS1 listed as a CS/E major at time of to the advanced-track CS1 course for impacts this might have had.
enrollment).
Contributing to Programmatic Retention are the number of 4. RESULTS
original CS/E majors who are retained as well as the number of We evaluate the retention of students in the CS1-BP course using
new majors (students who were not CS/E majors at the time of the metrics defined in Section 3.
CS1 who become CS/E majors 1-yr later). We define
programmatic retention as the sum of these forms of retention: 4.1 Retention of Students Who Pass CS1
Figure 1 provides overall 1-yr programmatic retention by gender
Original Majors Retention: (#of students listed as a and overall based on the number of CS/E majors who pass CS1.
CS/E major at time of enrollment in CS1 who are First we focus on the two components of Programmatic Retention:
majoring in and taking a computer science course one retained original majors and new majors. In terms of retained
year later) / (#of students in CS1 listed as a CS/E major original majors, CS1-BP increased retention from 68% to 81%
at time of enrollment) (p<0.01). Although both genders improved in the retention of
New Majors Retention: (#of students not listed as a original CS/E majors, males improved slightly more than females.
CS/E major at time of enrollment in CS1 who are in a Male improvement from 68% to 82% was statistically significant
CS/E course and CS/E major 1 year later) / (#of students (p<0.01) however the improvement for females from 68% to 79%
in CS1 listed as a CS/E major at time of enrollment) was not (p=0.058). A similar story appears for new majors. New
Measuring the addition of new majors to the program is difficult majors, as a component measured out of the number of initial
for the following reason: students may make the decision to majors, went from 3% to 8% overall. Males benefitted slightly
change major and enroll in CS/E classes but delay in officially more, experiencing a gain from 3% to 8% whereas females went
changing their major (as recorded). One potential measurement from 5% to 9%.
would be to infer that students enrolled in a CS/E class one year The New Major Retention metric allows for us to evaluate the
later have changed their major to CS/E but this could be contribution to programmatic retention from new majors, but may
problematic as it may capture students minoring in CS/E, taking hide details concerning from where these new majors come. Non-
more classes out of interest, etc. We instead take a conservative majors enrolled in our CS1 course vary widely by term and
approach and only count those students enrolled in a CS/E class consist of a variety of students, many of whom are taking CS1 as
one year later who have also changed their major to CS/E. We a requirement for a related major. As such, reporting the number
recognize that this may undercount new majors, but it does so to of new majors as a percentage of passing non-majors can be
both classes (CS1-Before and CS1-BP). Future work can more deceptively low as non-majors are often non-freshman in a
accurately measure major outcomes after graduation. required course for their major and are not “potential majors.”
We are most interested in programmatic retention for the However, even when evaluating this metric (new majors one year
following reason: we enroll a certain number of students in our later divided by CS1 non-majors) we find substantial
program—if they start in CS1, how many are still in our improvements for CS1-BP. For CS1-Before, 3% of passing non-
“pipeline” after a year? This number will include students we majors become majors 1-yr later for males, females, and overall.
167
Retained Majors New Majors CS1-before CS1-BP
100% 100%
Programmatic Retention
86%
80% 80%
80%
73% 71% 73%
71% 72%
70%
70% 68%
60%
60%
50%
50%
40%
CS1-BP
CS1-Before
CS1-Before
CS1-Before
CS1-BP
CS1-BP
40%
Males Females Overall
Figure 2. Programmatic Retention by gender for both versions of CS1
for a single instructor. Differences for Males and Overall are
statistically significant (p<0.05).
Males Females Overall As reported in Section 3, the pass rate for majors increases from
Figure 1. Programmatic Retention by gender for both versions of 80% in the original course to 94% in the new course. By
CS1. Includes retained majors and new majors. Differences in changing the denominator in the equations from Section 3 to be
Programmatic Retention are statistically significant (p<0.05). the number of computer science/engineering majors who are
enrolled after the deadline to withdraw from the course (week 4)
For CS1-BP, these numbers rise to 14% (p<0.01), 7% (p=0.046), instead of those who pass, we can further evaluate the impact of
and 11% (p<0.01) respectively. Although these improvements in these changes. This adds in those students who did not pass the
new majors are large (factor of 3.7 increase overall), the large course (earning a D or F grade, or withdrawing from the course).
variance in reasons for students taking the class limits potential Figure 3 provides these results. The impact of the new course is
conclusions. more pronounced as the two effects—an increased pass rate and
The primary result from Figure 1 is that of overall programmatic an increased retention rate—compound to increase the
retention. Although programmatic retention improves for both programmatic retention from 57% to 84% (p<0.01).
males and females, the improvement in male retention is slightly Improvements in male and female programmatic retention are
higher than that of females. Overall improvement in retention is statistically significant (p<0.01).
substantial: 18% (p<0.01). As a measure of improvement, 29% It was surprising to us to discover that in our CS-Before course
of programmatic majors were lost after one year, whereas only only 57% of the number of declared CS/E majors at the end of
11% were lost after revising the course. This represents a 62% week 4 persisted one year later. Although this is a derivative of
reduction in lost programmatic majors. the change in majors’ pass rate in the course (80% increasing to
4.2 Instructor Effect 94%), the view from the programmatic level is a bit more stark.
A common criticism of results for new course design is that those
most apt to change the course tend to be “good” instructors who,
4.4 Retention by Students Enrolled at Week 1
In Figures 1 and 3, we excluded students who start the course, but
of course, “do well.” To address this potential concern, Figure 2
drop it before week 4 (an event unrecorded on their transcript).
provides the programmatic retention for a single instructor who
For this work, we make the reasonable assumption that CS/E
taught the course four times in each of the two styles. These
declared majors enrolled in the course at the end of week 1 intend
results are comparable with the overall results, demonstrating a
to give the course (and perhaps computing) a serious try. As
14% increase in programmatic retention (p<0.01). The benefit is
another potential threat to validity, if more students drop the CS1-
again more substantial for males (15% p<0.01) than females
BP class than the traditional CS1 class in the first 4 weeks, one
(12%, not statistically significant p=0.08).
could again argue that CS1-BP artificially inflates the retention
4.3 Retention by Students Enrolled in CS1 and pass rates. We actually found the opposite effect had
occurred in that fewer majors dropped before week 4 in CS1-BP
(Pass, Fail, Withdraw) (2%) than in CS1-Before (11%).
The results thus far have focused on the percentage of students
retained relative to the number of majors who pass the CS1 Figure 4 provides the programmatic retention for each of the CS1
course. This could also be a potential threat to validity, as one variants for majors enrolled in CS1 at the end of week 1. The
might substantially reduce the number of students passing the denominator for numbers in this graph includes majors who pass,
course to artificially affect retention. However, in the case of this fail, withdraw, or drop after week 1. The impact of the new
new course, the number of students who pass increases as does the course is now magnified by three effects: fewer drops, fewer
number of those students who are retained. failures (withdraw, D, or F), and increased retention among those
168
CS1-before CS1-BP CS1-before CS1-BP
100% 100%
Programmatic Retention (Week 4)
70% 70%
59%
60% 57% 57% 60%
51% 50% 51%
50% 50%
40% 40%
Males Females Overall Males Females Overall
Figure 3. Programmatic Retention by gender for majors original Figure 4. Programmatic Retention by gender for majors original
enrolled in CS1 at Week 4. Differences are statistically significant enrolled in CS1 at Week 1. Differences are statistically significant
(p<0.05). (p<0.05).
who remain. These effects compound: increasing programmatic what about the experience students have coming in our “front
retention from 51% to 82% (p<0.01). Differences in both male door”? It is when we evaluate the number of initially enrolled
and female programmatic retention based on week 1 are also majors that we believe the true story is told. Previously, students,
statistically significant (p<0.01). perhaps with little idea of what computing is about and what one
does to “study” and “learn” computing, would start in our course,
4.5 Advanced Track CS1 but one year later only 51% of that number would still be studying
In Section 3 we discussed the potential threat to validity that our computing. By adopting a complementary set of best
major was impacted (admission to the major was more selective) practices, we seem to have patched up some major leaks in
during the timeframe of CS1-Before (2001-2008) and not- our pipeline—improving retention to 82%. In this section, we
impacted during the timeframe of CS1-BP (2008-2011). Those first revisit threats to the validity of this work and specifically
timeframes were also different in terms of national climate consider the impact on the female pipeline. We then discuss
regarding computing as a field. Lastly, students from those potential reasons why the trio of best practices has positively
different timeframes may have differed in terms of ability, interest impacted retention.
in CS, and perceptions of CS.
5.1 Threats to Validity
To inform us as to the effect of timeframe on our results, we The threats to validity from Section 3 remain relevant. The
evaluated another class at our institution which had not changed. student population in our courses could have changed due,
The course studied and modified here was for our standard CS1 perhaps, to perception of the computing field or to our varying
track designed for students without (much) programming admission standards (impacted status). Additionally, this post-hoc
experience. The alternative advanced CS1 track is designed for study reports data from 10 years of real course data. We have no
students with programming experience. We hypothesized that control over instructors and their very real and possibly varying,
national change in climate, change in students over time, and the effects. That said, we report our best efforts to provide the
shift of the major from impacted to not-impacted would benefit readers a chance to assess these threats for themselves. We find
both sections of CS1. Analyzing programmatic retention for the that an alternate CS1 at our institution did not experience a
advanced CS1 course, we found a minor improvement (83% to statistically significant increase in retention over the period of our
87%); however the result was not statistically significant (p=0.12). study. We also show results attempting to control for the
Although the advanced and standard CS1 tracks may have a instructor effect by providing a within-instructor comparison for
different composition of students, this result strengthens our belief the one instructor who has taught the course the most, covering
that the retention improvements for CS1-BP were from the both versions. We find similar gains for courses just taught by
changes to the course, rather than the timeframe. that instructor; although we do not analyze these courses at any
finer granularity for factors which might impact retention.
5. DISCUSSION
Our implementation of Media Computation, Pair Programming, 5.2 Gender Differences
and Peer Instruction appear to provide a proverbial hat-trick of In Section 4.1 we found that CS-BP had a larger beneficial impact
instructional design resulting in significantly increased on males than females. However, that section analyzed retention
programmatic retention. This increase comes from both the among students who pass the course. As previously mentioned,
retention of original majors and the addition of new majors. one might artificially increase retention by passing fewer students
These results include data from 10 years of instruction including (or by encouraging students to drop). After excluding these
26 different sections taught by 8 different instructors impacting effects in Sections 4.3 and 4.4, males now benefit slightly less
over 3,000 students. Of those students who pass the course, the from the new course as there is a 31% improvement for males
new course design increases retention by 18% (71%-89%). But versus a 32% improvement for females. This effect is much less
169
substantial than the difference in 4.1 and occurs because female situ study, we evaluate retention for students who pass the CS1
pass rates improve in CS1-BP over CS1-Before more than males course and find an 18% absolute improvement in programmatic
(female 15% and male 13%). Thus, evaluating retention based on retention (students in the major one year later). Although these
week 1 and week 4 enrollments provides a potentially more numbers summarize results from multiple instructors, we also find
accurate picture than retention among those students who pass. a 14% improvement for an individual instructor having taught
multiple terms both with and without these improvements. Lastly,
5.3 Benefits of Best Practices we find that failure rates and drop rates decreased in the best
As previously mentioned, all three best practices—Media practices course. The combination of fewer students dropping,
Computation, Pair Programming, and Peer Instruction—were more students passing, and more passing students retained
selected because of the body of research supporting their benefits. resulted in an absolute improvement of 31% (from 51% to 82%)
Here we wish to provide our intuition as to why these best in programmatic retention among students initially enrolling in
practices, combined, may provide increased retention. our CS1 course. In this work we discussed our rationale for
adopting these changes, potential threats to the validity of our
A key study of why students are not retained in the sciences,
results, and how these three best practices synergistically combine
They’re Not Dumb, They’re Different [12], may provide insight
to improve retention.
into this course. In that study, non-science graduate students and
a non-science Ph.D. enrolled in introductory science classes and 7. Acknowledgements
reported their experiences. From our analysis of the work, there Thank you to Dr. Rachel Roe-Dale and Dr. Jaime Spacco for their
are four key themes relevant to computer science that stem from assistance with this work. We would also like to thank our
Tobias’ subjects’ impressions of university science classes that anonymous reviewers for their constructive feedback. This work
make them unappealing: Context – science classes lack was supported in part by NSF grants 0933635 and 1140731.
relevance to the real world; Engagement – student comments are
not valued and they are uninvolved in lecture; Competition – 8. REFERENCES
students are in adversarial relationships with each other for [1] Crouch, C. H., and Mazur, E. Peer instruction: Ten years of
grades; and Community – students are on their own without experience and results. American Journal of Physics 69.
support. We believe the trio of best practices addresses each of 2001.
these concerns: [2] Cutts, Q., Esper, S., and Simon, B. Computing as the 4th
Context. Media Computation provides the highly relevant and "R": a general education approach to computing education.
familiar domain of images and sounds for students. Moreover, the In Proceedings of the seventh international workshop on
homework assignments designed for the course allow for a level Computing education research. 2011.
of creativity in the solutions (photo collages, etc.) which might be [3] Forte, A., Guzdial, M. Motivation and nonmajors in
difficult in different domains. computer science: identifying discrete audiences for
introductory courses. IEEE Trans on Education, 48(2). 2005.
Engagement. Peer Instruction involves students directly into [4] Guzdial, M. and Ericson, B. Introduction to computing and
lecture in that they are frequently solving problems and discussing programming with java: a multimedia approach. Prentice-
the material with other students and/or the professor. The Hall. 2007.
applicable contextual domain from media computation also helps
[5] McDowell, C., Werner, L., Bullock, H. E., Fernald, J. Pair
improve student interest and, hence, engagement.
programming improves student retention, confidence, and
Competition. Our changes result in student grades no longer program quality. Commun. ACM 49, 8, 90-95. 2006.
being entirely individualistic, as parts of students’ grades are now [6] Porter, L., Bailey-Lee, C., Simon, B., Cutts, Q., and Zingaro,
based on cooperative efforts. Homework assignments and Labs D. Experience report: a multi-classroom report on the value
are done as pairs using Pair Programming. Class participation is of peer instruction. In proceedings of the 16th Annual
graded on their answering of questions, not correctness. Peer Conference on Innovation and Technology in Computer
Instruction has them working in cooperative groups during class. Science Education. 2011.
As such, the environment is much more collaborative. [7] Porter, L., Bailey-Lee, C. Simon, B., Zingaro, D. Peer
instruction: do students really learn from peer discussion? In
Community. The same elements that reduce competitiveness the 7th Annual International Computing Education Research
also foster community. Pair Programming in assignments and Workshop. 2011.
labs provides students support avenues in the form of classmates. [8] Simon, B. http://www.peerinstruction4cs.org/
Peer Instruction provides each student with a group of other [9] Simon, B., Kinnunen, P., Porter, L., Zazkis, D. Experience
classmates with whom they work in each lecture, potentially report: CS1 for majors with media computation. In Proc. of
creating an out-of-class support group as well. 15th Annual Conference on Innovation and Technology in
In summary, one can see how Media Computation, Peer Computer Science Education. 2010.
Instruction, and Pair Programming facilitate an engaging and [10] Simon, B., Kohanfars, M., Lee, J, Tamayo, K., and Cutts, Q.
cooperative classroom where students work with a domain Experience report: peer instruction in introductory
relevant to their experiences. computing. In Proceedings of the 41st SIGCSE technical
symposium on computer science education, 2010.
6. CONCLUSION [11] Snyder, L. Barnes, T., Garcia, D., Paul, J., and Simon, B.
In the fall of 2008 we adopted a trio of CS1 best practices The first five computer science principles pilots: summary
including Media Computation, Peer Instruction, and Pair and comparisons. ACM Inroads 3, 2, 54-57. 2012.
Programming. These were chosen because of supporting [12] Tobias, S. They're not dumb, they're different: stalking the
evidence from computing education research literature. In an in- second tier. Research Corporation, Tucson. 1990.
170