Research Overview

An e-Portfolio Design Theory Supporting Ownership, Interaction, and Ease of Use

Nathan Garrett, Claremont Graduate University
http://conversation.cgu.edu/garrettn/
June 6, 2008

Abstract
The goal of this dissertation project is to create and evaluate the effectiveness of an ePortfolio
system supporting student ownership, peer learning, and ease of use. Such a system is
hypothesized to increase users’ intrinsic motivation, improve student learning, and yield reliable
and valid data for assessment. It deals with increasing demands for external accountability by
demonstrating ways to make learning social, engaging, and demonstrable.

Research Question
The central concern of this project is the development of electronic portfolio design theory. How
does a portfolio change as it moves from pieces of paper to just another record in a centralized
database? How do system designers ensure their software supports learning while still generating
assessment data? Do current systems support the proper use cases?

My research goal is to advance portfolio design theory by evaluating the impact of ownership,
ease of use, and peer learning in predicting user satisfaction. Building upon previous work using
social software in education, I have developed a number of design features to support these
constructs. My research approach is to create a software artifact, deploy it in an educational
setting, and use survey and usage data to evaluate my design features and the overall design
model.

Reason for Research

As universities react to the Spellings Commission’s call for increased comparability and external
accountability, portfolios have begun to look like the “golden tool” for assessment. Portfolios
provide a rich dataset that can show critical thinking, growth over time, and deep learning
(Barrett, 2005). However, as portfolios are increasingly adopted for assessment purposes, their
original pedagogical purpose is often subverted (Barrett, 2004). Instead of using portfolios for
learning, and their contents for assessment, portfolio initiatives and software are often solely
focused upon generating data for accreditation bodies.

As a result, most ePortfolio systems tend to be large, centralized, and standardized. The need to
minimize demands on faculty time has pushed software like OSP and others into rigid checklist
and “fill in the blank” designs. Software like LiveText and Iwebfolio encourage administrators to
lock down student portfolios to ensure consistency. In Iwebfolio, these controls include limiting
the ability to change page names, add new pages, or choose a non-standard template. Portfolios
are private by default, and students must go though a tedious process for each person they would
like to give access.

1
Meanwhile, researchers are trying to figure out ways to make portfolios more “sticky” (Jafari, A.,
McGee, P., & Carmean, C. 2006b) so that students will be able to see them as the learning
opportunity they represent. Professors want to use portfolio software to help students reflect,
share, and develop meta-cognitive abilities.

Studies of students have shown that they want education software that helps them to connect
with each other, express their individuality, and is easy to use (Jafari, A., Mcgee, P., & Carmean,
C, 2006a). Frequently, faculty and staff use free social software on the Internet to do all these
things, finding it easier to use and more fulfilling than portfolio software. As one faculty member
says:

“I engage in many e-portfolio-like practices. Those involving dedicated e-

portfolio tools have been far less satisfactory than those involving social software
tools such as blogs, wikis, or social networking sites.” (Becta, 2007, p.24)

Literature Review
Almost all current ePortfolio software packages are driven by a single dominant design model.
This paradigm defines a portfolio as a database of individual artifacts organized into a number of
different views and submitted to specific audiences. This model is well suited for controlling
access, providing multiple views of data, and providing assessment data. However, it makes
portfolios complex to construct, and the permissions settings discourage sharing and peer
learning. Unfortunately, this paradigm is so common that almost all major programs use it. As a
result, Cohn and Hibbitts (2004) identify the “ossification of the current prefabricated, one-size-
fits-most portfolio” (p.2).

My epistemology for this project is primarily based upon constructivism. In particular, much of
this project has been inspired by Schön’s concepts of ‘learning to be’ and reflection (1982, 1986).
Schön’s concept of reflection built upon the work of John Dewey, who proposed reflection as a
mental version of the scientific process (Dewey, 1933). Dewey defined reflection as a set of
mental thought experiments. To reflect, a learner must become open to the idea of uncertainty, or
not knowing something. They then evaluate possible approaches to structure the problem, and
evaluate potential outcomes based upon evidence and rational thought. Learning portfolios are
an excellent way to induce reflection (Moon, 2001). The connection between reflection and
learning are discussed in depth in Moon (2001), showing how reflection creates the right
conditions for learning.

A portfolio is a fundamentally collaborative work. Zubizarreta (2006) argues that portfolios

require three domains of activity: documentation, reflection, and collaboration. Collaboration
should include faculty members (classroom teachers or an advisor) and other students. As he
says, “… reflection is facilitated best by not leaving students individually to their own devices in
thinking about their learning but by utilizing the advantages of collaboration and mentoring in
making learning community property.”

Unfortunately, most portfolio software systems do not provide robust features for collaboration.
A survey conducted in the United Kingdom (Becta, 2007) found that only a third of the surveyed

2
students in the case studies agreed that their system “shows me what my friends are learning”
(p.17).

Adoption of ePortfolios can be difficult. Goldsmith (2007) found that usefulness and ease of use
were the most important predictors of student adoption. Surveys of students have found that they
want simple education software that helps them connect with each other and express their
individuality (Jafari, A., Mcgee, P., & Carmean, C, 2006a). Students frequently request systems
more like the social software they use outside of school (Jafari et al., 2006a). As one student
says, “social software has been around for a while now and it’s a lot more user-friendly” (Becta,
2007, p.16).

Social software supports self-governed, problem-based and collaborative activities (Dalsgaard

2006). Because social software is an emerging area, its definition is still fluid. Unsurprisingly,
some of the best (or at least most faithful) definitions have been created through social software
tools. The social software page on Wikipedia says that “[s]ocial software enables people to
rendezvous, connect or collaborate through computer-mediated communication” (Wikipedia
2007). Social software includes a host of different technologies, including blogs (such as
Blogger or Xanga), wikis (Wikipedia), social bookmarking (del.icio.us), podcasting (iTunes) and
picture sharing (Flickr).

Previous educational research on social software has focused on defining student characteristics,
social software identifying properties (e.g. web 1.0 versus web 2.0), and appropriate pedagogical
uses, and has often resulted in recommendations for high-level strategic implementations
(McAllister et al. 2007). Unfortunately, many academic researchers focus on identifying and
describing emerging practices, rather than rigorously identifying their effectiveness and impact
(McAllister et al. 2007).

The United Kingdom’s lead agency for information and communication technology in education
has found great potential to make connections between e-portfolio processes, such as storing,
reflecting and publishing, and learners’ use of emerging social software tools used outside formal
education (Becta 2007). In addition, McAllister et al. found that social software can provide both
formative and summative assessment (2007).

Theoretical Basis
This research project uses a design science approach. Design science builds upon Herbert
Simon’s concept of the science of the artificial. Rather than attempting to theorize and justify,
the cycle common to natural science research, I attempt instead to build and evaluate an IT
artifact. My artifact will then act as an alternative model (or mindset) for the creation of
portfolio software to support higher education. Design science research precisely includes this
kind of theorizing (March and Smith, 1995).

Following Walls, Widmeyere and El Sawy (1992, 2004), I establish important requirements for a
portfolio system and then connect those requirements to specific features. Not explicitly
mentioned are the standard website design requirements, such as automated backups and a fast
page response time (including level 1 and level 2 requirements from Zhang. 2007). These are all
critical dimensions, but ultimately are common to all online systems. Because a web portfolio

3
system builds upon the design requirements for web systems, those requirements can be left out
of this particular design theory.

Ownership
The concept of ownership is central in portfolio systems. As stated earlier, the defining
characteristic of a portfolio is that it is owner–centric. Unfortunately, the focus on assessment has
shifted portfolio software into assessment management systems (Barrett, 2004) that are
institution-centric. Returning the focus to the student suggests a number of design features.

First, material should be located in the individual’s space, and then submitted to a class or
program as a final step. Material should be organized around a person, and not necessarily a set
of learning goals or courses.

Second, students should have full control over their portfolios. In many systems, this principle is
violated by locking the portfolio while it is being assessed, instead of simply making an off-line
for future assment. Students should also not be forced to develop their entire portfolio in the the
structure of the assessment criteria. While users need to satisfy course and program
requirements, the approach should be up to them.

Third, users should be able to choose a visual presentation style of their own. Their site should
reflect their own personality, and not that of the school. Commercial companies selling portfolio
software have the key selling point of user’s ability to select and present evidence in their own
style (Beetham, 2005). However, many packages available today have limitations on the way
content can be arranged. As an example, OSP forces students into a single (very long) page for
their finished portfolio.

Peer Learning
The collaborative nature of portfolios has been poorly acknowledged. As one of the 3 basic
realms of activities proposed by Zubizarreta (2006), it should be reflected in systems’ design as
more than an afterthought. This leads us to a number of design features.

First, casual exposure to peers’ content should be one of the basic use cases of any portfolio
software. Most systems emphasize security and privacy, and, as a result, leave out the casual
learning that occurs in a traditional paper classroom portfolio project. One of the major unique
features of this project is in its attempt to provide support for peer learning, recognizing the
usefulness of casual peer content exposure.

Second, pages should be public by default. While the user has ultimate control over the ‘public-
ness’ of their portfolio, making the default open will bias the system towards collaboration.

Third, website addresses should be short and easy to communicate. Students should be able to
quickly access their portfolio without needing a long and complex address.

4
Ease of Use
One of the major complaints about existing systems is that they are difficult to learn and use.
Research into use adoption has found that the Technology Acceptance Model (Davis, 1989) well
describes students’ acceptance of portfolio software (Goldsmith, 2007). Ease of use is a critical
component leading to adoption.

First, loading content into the system should be as easy and as quick as possible. Allowing
embedded content for other popular online services (such as Google Video or Flickr), and
making the process as simple and fast as possible is crucial.

Second, editing a page’s content should be as simple as possible. Modern GPL AJAX javascript
libraries (like Yahoo’s) make it possible to create drag & drop pages that can be edited in place.
This should make it possible to easily and quickly load either new content, or to re-use that
which has already been uploaded.

Third, the most common portfolio style uses a collection of pages that are then organized for
specific snapshots. This assumes that students will create a large amount of re-usable content
that is submitted a number of times. This pattern is not generally found in social software.
Many systems spend so much effort supporting re-usable content that they introduce unneeded
complexity. The ability to re-use pages is a minor feature, especially when considering how easy
it is to simply copy and paste content.

Workflow Cycle
Ultimately, every portfolio system needs to support the basic workflow cycle of collect, select,
reflect, and assess. Collect refers to the process of saving material used to satisfy course- related
requirements. Select then refers to the process of choosing from among the saved artifacts to find
those illustrating either achievements or learning. After selecting a set of artifacts, students then
reflect on their learning, using their artifacts and a set of learning objectives as a prompt. Lastly,
the teacher (or staff) assess the students’ learning and record the results for further use. This
cycle then repeats for the next assignment or class, or terminates at the end of the student’s
enrollment.

Research Design
The Social Learning Lab at Claremont Graduate University has used Elgg as our primary social
learning platform for two years. Elgg provides users with a blog, file gallery, avatar, and
personalized templates. It allows individuals to form academic and non-academic communities,
and has a finely grained, user-controlled, permission system.

I developed a wiki-like collaborative editing tool that has been essential in supporting both our
school’s classes and a set of cross-disciplinary courses. After I complete the full implementation
of my design model in the next version of this plug-in, it will be used as the required portfolio
system for the School of Information Systems and Technology beginning in Fall 2008.

5
Currently, my tool is used by a number of other organizations, such as Emerald Publishing,
Brighton University, and Oxford University. As a result, I receive informal feedback though a
number of sources. Formal data will be gathered through site usage analysis, surveys, and web
server logs.

The pre-test will be used to assess users’ interest in portfolios before they use the application. It
will be short, using a modification of the TAM (Davis, 1989) questions for assessing the
perceived usefulness of the software to the user. This will be useful in analyzing a user’s
motivation before experiencing the software, and for comparing against their level of perceived
usefulness after submitting the portfolio.

The post-test will be used after the members of each group have submitted their final portfolio.
The first half of the test will be as close to the Zhang (2007) instrument as possible to allow for
valid comparisons. The second half will contain additional questions to measure the user’s
feelings of ownership, ease of use, and perceived usefulness

Hypotheses
This project’s primary hypothesis is that ownership, peer learning, and ease of use are predictors
of student satisfaction and motivation. The evaluation of the effectiveness of my design features
also suggests a number of secondary hypotheses.

First, I predict that users’ ability to organize, control the contents, and create and/or choose a
template for their portfolio will increase their feeling of ownership.

Second, I will measure the impact of the new peer learning features. The major way that the
feature can be assessed is simply to evaluate how much it was used. If people frequently look at
other students’ pages, then I can assume they found it useful. Student self-reports will also be
valuable in showing the extent to which they found looking at peer work valuable.

Third, I have made a significant number of design decisions that should improve ease of use.
Using Zhang (2007) as a benchmark, I will determine the overall ease of use, as well as target
specific areas that work well or poorly.

6
Work Cited
Jafari, A., Mcgee, P., & Carmean, C. (2006a). Managing Courses & Defining Learning: What
Faculty, Students, & Administrators Want. Educause Review July/August. Retrieved
October 8, 2007 from http://www.educause.edu/ir/library/pdf/erm0643.pdf.

Jafari, A., McGee, P., & Carmean, C. (2006b). A Research Study on Current CMS & Next
Generation eLearning Environment [White paper, electronic version]. Retrieved October
8, 2007 from Next Generation Course Management System Group website
http://ngcmsgroup.epsilen.com/Groups/Public/Home.aspx?pr=ngcmsgroup.

Barrett, Helen C. (2004) Differentiating Electronic Portfolios and Online Assessment

Management Systems. SITE 2004 Conference Proceedings.

Barrett, H. (2005). Researching Electronic Portfolios & Learner Engagement [White paper,
electronic version]. Retrieved October 8, 2007 from The Reflect Initiative, TaskStream
Inc website http://electronicportfolios.com/reflect/whitepaper.pdf.

Becta (2007). Impact Study of E-Portfolios on Learning [White paper, electronic version]. June
2007. Retrieved April 11, 2008 from Becta website http://partners.becta.org.uk/upload-
dir/downloads/page_documents/research/impact_study_eportfolios.pdf

Cohn, E., Hibbitts, B. (2004). Beyond the Electronic Portfolio: A Lifetime Personal Web Space.
Educause Quarterly 27(4). Retrieved April 14, 2008 from
http://connect.educause.edu/Library/EDUCAUSE+Quarterly/BeyondtheElectronicPortfo/
39884.

Dalsgaard, C. (2006). Social software: E-learning beyond learning management systems.

European Journal of Open, Distance, and E-Learning. Retrieved October 8, 2007 from
http://www.eurodl.org/materials/contrib/2006/Christian_Dalsgaard.htm.

Davis, F. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of
Information Technology. MIS Quarterly September 1989

Dewey, J. (1933). How We Think (Second Edition). Boston, MA: D.C. Heath and Company

Goldsmith, D. (2007). Enhancing Learning and Assessment through E-Portfolios: A

Collaborative Effort in Connecticut. New Directions for Student Services 119(Fall) 32-42.

McAllister, A., Johnston, P., Evans, V., Larri, L., & Stolz, S. (2007). Networks, Connections and
Community: Learning with Social Software [White paper, electronic version]. Retrieved
October 8, 2007 from Australian Flexible Learning Framework website
http://www.flexiblelearning.net.au/flx/go/pid/377.

Moon, J. (2001). PDP Working Paper 4 – Reflection in Higher Education Learning. [White
paper, electronic version]. Ltsn Generic Centre: Learning and Teaching Support Network

7
Salzman, S., Denner, P., Harris, L. (2002). Teacher Education Outcomes Measures: Special Study
Survey. Annual Meeting of the American Association of Colleges for Teacher Education.
February 23-26, 2002, New York, NY.

Schön, D. (1986). Educating the Reflective Practitioner. San Francisco, CA: Jossey-Bass Inc.

Schön, D. (1982). The Reflective Practitioner. New York, NY: Basic Books, Inc.

Zubizarreta, J. (2006). The Learning Portfolio: Reflective Practice for Improving Student
Learning. Boston, MA: Anker Publishing Company, Inc.

Zhang, S (2007). Model, Design and Efficacy of Next-Generation Eportfolio Systems (Doctoral
dissertation, Claremont Graduate University, 2007).