EDIT7350E eLearning Evaluation & Assessment | Dr.

ChanMin Kim | Spring 2017

Evaluation Report

UGAs Center for Teaching and Learning eLC Support Page

Evaluator Team #3
Jalonna Kearney, Lana Kempton, and Davina Loskot
May 1, 2017
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

1. Introduction and Background

The main goal of the Center for Teaching and Learning (CTL) at the University of
Georgia is to develop support materials and provide valuable assistance for faculty,
graduate teaching assistants and students on the ever-changing online instructional
technology. The CTLs help webpage, http://www.ctl.uga.edu/elc provides eLearning
Commons (eLC) learning materials designed to assist users in the form of videos,
printables, and other various step-by-step tutorials.

The evaluators have been asked to survey faculty and make a formal
recommendation on how CTL can rebuild, organize and streamline its help webpage in
order to make it more user friendly and easier to navigate.

The evaluators of the project are:

Lana Kempton, an Instructional Technologist in Macon, Georgia, pursuing her
Masters in Education
Jalonna Kearney, a Math Intervention Teacher in New Orleans, Louisiana,
pursuing her Masters in Education
Davina Loskot, a String Orchestra Director in Lawrenceville, Georgia, pursuing
her Doctorate in Music Education.
All evaluators are currently enrolled in the EDIT 7350E class at the University of
Georgia.

2. Evaluation Purpose

The primary purpose of this evaluation was to:

1) Conduct a short survey of the targeted audience on the usefulness and accessibility
of eLC learning materials, and
2) Make recommendations on how to rebuild, organize and streamline the CTLs help
webpage in order to improve its accessibility and usefulness.

3. Stakeholders

The Primary Stakeholders in this evaluation are:

Dr. Sherry Clouser, Assistant Director of Learning Technologies at the Center for
Teaching and Learning at the University of Georgia and the Faculty and Teaching
Assistants at the University of Georgia.
The Target Audience: The faculty as well as teaching assistants of the University of
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

Georgia is also a stakeholder in this evaluation since the final changes suggested will
directly affect the usefulness and accessibility of the help webpage.

Table 1. Stakeholder Information

Stakeholder Importance Perspective of Role in the Evaluation
Level Interest

Dr. Sherry Clouser, Primary The usefulness The Evaluation Team

Assistant Director of and accessibility conducted an interview
Learning of the eLC with Dr. Sherry Clouser.
Technologies at the learning materials Dr. Clouser has also
Center for Teaching offered to help send out
and Learning, The the survey that the team
University of has constructed, and
Georgia forward any help desk
ticket data that will be
collected.

The faculty and Primary and The usefulness A survey will be sent out
teaching assistants Secondary and accessibility to various faculty
of the University of of the eLC members to collect data
Georgia learning materials on the usefulness and
accessibility of the eLC
learning materials. The
targeted audience will
be the faculty of the
University of Georgia.

4. Logic Model

Context
The center of teaching and learning at UGA wants to seize the opportunity to
update their faculty support page for eLC prior to the release of a major eLC update that
will change the look and feel of the system.

Population and Needs

Faculty need 24/7 support for the eLC system. The current guides and resources
on the support page will become outdated once the eLC update is complete. Revised
versions of these guides and resources need to be produced.

Age and Phase

The current documentation that has been utilized for the past 5 years will soon
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

need updates. These updates will be completed prior to Fall 2017.

Resource/Inputs
The center for teaching and learning has staff working on resolving this issue.
The staff has computers along with the software and hardware necessary to complete
the task. The staff also have access to faculty, evaluators, and web designers to assist
with the task.

Table 2. A Simplified Logic Model in a Table Form

Context
The eLC system will be updating to a new interface within the next academic year.
This will render the current documents on the help page out of date. This evaluation
will conclude what current topics should be utilized in the future and what may need
to be added to improve usefulness and usability of the site.

Population and Needs

UGA Faculty need a 24hr resource available to them in order to answer common
eLC questions relating to the management of their course(s) as well as answer
student questions.

Age and Phase

The current CTL page services the eLC version that has been in place for 5 years
at UGA. The revised eLC page will be implemented over the course of Fall of 2017
to accommodate for aesthetic updates in the eLC Daylight update.

Resources/Inputs Activities Outputs Short-Term Long-Term

Outcomes Outcomes

Human resources: Determine what Common Faculty will Faculty will

Learners, common eLC question list be able to be able to
Evaluators, questions are self support apply their
Designers, CTL Updated themselves knowledge of
Staff Develop resources with the eLC to their
resources and and guides guides and courses
Technology: faculty guides resources
Computers, to assist with Updated available on Faculty will
Software, common eLC CTL eLC the webpage be able to
Hardware questions webpage increase their
Increase the quality of
Develop a System for number of instruction at
website that answering Faculty using UGA.
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

hosts these questions in eLC

guides and place The quality of
resources so Increase student
that they may Fewer help confidence of learning will
be available for tickets the current increase at
faculty use submitted faculty using UGA
eLC
Develop a
method for Provision of
faculty to ask support for
for additional faculty 24/7
assistance with
eLC issues

Activities
First, common eLC questions will need to be compiled. Resources and guides
will be developed to aid with these common questions. These guides will be made
accessible through the CTL eLC webpage. Finally, a system for faculty to ask questions
will be installed.

Outputs
A common question list will be produced as well as resources and guides to
match each question. The CTL eLC website will be updated and a feedback and
question service will be made available to faculty.

Short-Term Outcomes
The faculty will become more self sufficient in answering their questions using
the resources and guides provided. The availability of 24/7 help will make more faculty
confident that they can effectively utilize the system and will encourage additional
faculty to begin using eLC.

Long-Term Outcomes
Faculty will be able to apply their acquired knowledge to their future courses.
Being more knowledgeable in the eLC system may allow faculty to increase their quality
of teaching and in turn increase the quality of their students learning.
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

5. Evaluation Questions

We expect that the questions to be investigated will give us insight into the
effectiveness of the eLC support site. This evaluation will address the following
concerns of stakeholders and attempts to meet the need of updating the Center for
Teaching and Learning eLearning website.

1) Is there any high frequency content missing from the eLC page?
a) Are users able to find the information they are looking for?
b) Do users submit tickets for similar topics?
i) Would a categorizing/logging method for ticket sorting be useful?
2) What is the overall satisfaction of UGA faculty with the eLC page?
a) Are users able to quickly and easily find the appropriate information?
b) If users have to submit a Help Center ticket, is a response received in an
optimal time frame?
3) What user feedback has already been given to managers about the eLC page?
a) Is there evidence that user feedback given during the 2016-2017 school
year has already been implemented into the eLC Help website?
b) Have any tickets submitted during the 2016-2017 school year already
been resolved?

Data gathered in response to these questions will provide the necessary information to
the client and the team to make decisions that positively impact the usability and
effectiveness of the eLearning website.

Table 3. Focus Areas, Questions, Design, and Rationale of Your Evaluation

Stakeholder Stakeholder Evaluation 3-5 Evaluation Evaluatio
Needs Focus Evaluation Design n Design
Areas Questions Rationale

UGA Center Determine Resources 1. Is there Pre-post In order to

for teaching the staff inputs any high one group gain the
and Learning effectiveness frequency comparison most
faculty of the Center Activities content insight into
for Teaching Download missing the
and Learning and sort from the stakeholde
(CTL) through eLC page? rs needs,
eLearning excel a pre-post
Website with spreadsheet 2. What is survey is
regards to of DHC the overall the most
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

providing tickets satisfaction efficient

troubleshooti of UGA method of
ng Contact eLC faculty with data
information web admin the eLC collection.
to faculty page? The Pre-
and staff Analyzing post one
Dr. Sherry about survey 3. On group
Clouser - managing results average per compariso
Assistant their eLC (specific day, how n allows
Director courses. questions?) many users for direct
Administrative and have used correlation
interview the eLC and
some page so far analysis of
faculty during the changes
2017-2018 made to
Analyze school the
feedback year? eLearning
submitted support
through eLC 4. What website.
page; user
feedback
Outputs has already
# of been given
satisfactory to
ratings on managers
post survey; about the
eLC page?
Outcome
faculty
satisfaction

6. Logistics and Timeline

The team has broken down the tasks displayed in Table 5 between the team
members. Lana will be leading evaluation question 1, Davina will be leading 2 and 4,
and Jalonna will be leading number 3. The team will be in regular communication
throughout the process via online meetings and text messaging. The documents
created throughout the evaluation process will be a group effort. Upon approval of the
evaluation plan first with Dr. ChanMin Kim and subsequently with Sherry Clouser
implementation will begin. After the evaluation period is complete an Evaluation Report
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

will be submitted to Dr. ChanMin Kim for revisions and upon completion of a final
version it will be turned into the client, Sherry Clouser. Please see the spreadsheet
below for the project timeline and estimated time allotted to each task.

Table 5. Project Calendar and Task Time Estimates

7. Limitations

This evaluation was specific to the University of Georgias Center for Teaching
and Learning (CTL). For the most part we received very positive responses from our
surveys and the data we collected however, there were a few limitations that occurred
during the implementation of our plan. The first limitation was the time we were given to
complete our evaluation. We believe that given more time, we could have opened our
survey to a larger percentage of the Faculty, Teaching Assistants and Student body and
included more responses. A second unforeseen limitation was the lack of response from
the Faculty and Teaching Assistants that we invited to inview. As we developed our
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

Evaluation Plan we also became aware that it is not plausible to include a link on the
eLC help page that will provide an answer for every question that might be proposed by
a user of eLC.

8. Data Collection and Analysis Plan

We utilized diverse data collection methods and sources in order to gain the most
complete insight into the needs of the stakeholder. We utilized an online survey,
eLearning support website statistics, and faculty-to-faculty e-mail correspondence.

Online Survey

We did not have direct access to evaluation participants such as UGA faculty
members so we gathered data via an online survey produced in Google Forms. Our
interview with Dr. Clouser suggested that we should utilize her access to the UGA
Teach listserv in order to increase the number of possible participants in the survey. We
allowed the participants two weeks to complete the survey. Participants used their own
devices to complete the survey within the allotted time. The survey consisted of 6 the
Likert-type scale questions and 2 open-ended questions:

Figure 1: Center for Teaching and Learning (CTL) eLC Support PAge Online Survey
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

The answers to these questions

provided insight to evaluation
question #2 [see Table 3].

D2L Help Center Tickets

Visitors to the eLearning page are
encouraged to utilize one of the
three assistance options:
submitting DHC Tickets, directly
contacting CTL, or utilizing Central
IT service to get additional
support. We analyzed the
information gathered by these
inquires to identify any possible
trends to address Evaluation
Question # 1.

E-mail Correspondence
Seasoned UGA faculty have a level of comfortability with Dr. Clouser and may utilize
standard e-mail to communicate any issues they are experiencing with eLC. We
analyzed specific emails directly relating to issues that may impact the effectiveness of
the eLC support site and used the results of our findings to help support any trends
discovered while investigating each Evaluation Question.

Table 4. Evaluation Data Collection and Analysis Plan

Evaluation Questions Data Data Source Activities

Collection
Method

Is there any high DHC Tickets DAT-L Weekly Download and sort
frequency content Support Logs through excel
missing from the eLC (existing) spreadsheets
page?
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

What is the overall Survey + Faculty and other analyzing survey

satisfaction with the Interview elc Support page results (specific
eLC page? visitors questions?) and
(developing) interview some faculty
Specific faculty
(low, mid, high)

What user feedback Central IT/CTL Sherry Clouser's Analyze feedback

has already been given Tickets email logs/Cental submitted through
on the eLC page? IT(existing) eLC page

9. Results

Over the duration of 1/1/17 to 3/24/17, 31 DHC (D2L Help Center) tickets were
submitted. Ten of these were submitted by faculty, 20 by students, and one by a CTL
staff member. Seventeen of these 31 (54.8%) tickets collected were common questions
that the CTL page has the potential to resolve. Seven out of the 31 (22.5%) could have
been resolved by current material on the CTL page. Four topics have been identified as
potential items to be added to the CTL page: I am having trouble logging in. Who can
help?, How to import quizzes, How to grade quizzes, How to release grade items &
date restrictions.

Graph 1. DHC Tickets Submitted by User Type

 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

Over the duration 6/14/2016 to 4/03/2017 378 Central IT/CTL Tickets regarding
the eLC were submitted. The Graph Below shows how these tickets are broken down
into more generalized categories. The Login/Access Issues is the largest category,
followed by the Miscellaneous category. The Miscellaneous category was made up of
the tickets that were individualistic and required case by case assistance, for example:
a) submitting application materials online for the Art Department, b) a cross listed
course no longer showing up in the eLC and c) password issues.

Graph 2. Central IT/CTL Tickets Submitted by Category

 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

Over the duration of 4/11/2017 to 4/15/2017, an online survey regarding

improving the overall effectiveness of the eLC Support Page was administered to
members of a UGA listserv e-mail group. The bar graph below shows the results of all 9
survey respondents.

Graph 3: Online Survey Results by Participant

 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

The pie chart below displays the survey results analyzed to reflect the level of
overall satisfaction of the users. The percentages displayed in Graph 4 show the
mathematical equivalence to the number of points scored out of 30 in the survey.

Graph 4: Level of Satisfaction

Unfortunately, no interviews were able to be conducted by the Evaluator Team.

Attempts made to solicit participants were unsuccessful and no responses were
received even after direct correspondence with participants.

10. Discussion

The collection of DHC tickets revealed that four topics could potentially be added
to the CTL eLC help page. These topics include: I am having trouble logging in. Who
can help?, How to import quizzes, How to grade quizzes, How to release grade
items & date restrictions. It was found that many of DHC tickets were specific questions
that would be best answered by the DHC and not the CTL eLC page. For example,
many questions were related to inquiries about if the eLC had specific capabilities or
questions related to an individual class scenario. Thus, the majority of inquiries are
being directed appropriately. It was found that only 22.5% of inquiries have the potential
to be answered by the current CTL eLC page. If the suggested additions are in fact
made, that percentage would rise to 54.8%. These additions would ensure that
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

potentially twice as many inquiries from eLC users could get the assistance they need
instantly from the CTL eLC page without having to contact the D2L help center where it
could potentially take days to get their questions answered.

The collection of the Central IT/CTL Tickets revealed that the most common
questions fell into the following categories: 1) Designing eLC Classes 13.8%, 2)
Uploading Content 9.5%, 3) Quizzes 10%, Login/Access Issues 27.2%, &
Miscellaneous 25.5%. The Login/Access Issues is the largest category, followed by the
Miscellaneous category. The Miscellaneous category was made up of the tickets that
were individualistic and required case by case assistance, for example: a) submitting
application materials online for the Art Department, b) a cross listed course no longer
showing up in the eLC and c) eLC password issues. As mentioned above, quite a large
number of the Central IT/CTL tickets were specific questions that would be best
answered by the DHC and not the CTL eLC page, for example: a) clicker issues and b)
b) a projector screen not responding to the touch panel, but door-side buttons work.

The information collected with the online survey revealed that many users enjoy
the functional categories topics are broken down into. The step-by-step tutorials are the
most helpful for first-time Desire 2 Learn (D2L) users. The level of instruction is
beneficial to novice users. Unfortunately, users who are more experienced with d2L find
the current instructions lack the necessary detail to help with their issues. The survey
participants reported difficulty in being able to find detailed guidance or information of
specific topics, such as how to track the output of students taking a quiz created outside
of eLC. This lack of resolution led to their submission of DHC tickets or Central IT/CTL
tickets. Although just over the majority of users do find the page extremely helpful, some
overall changes in the display and navigation of the page would be appealing to page
visitors. The consensus of the survey participants was that the current page seems
clunky and overwhelming.

11. Recommendations

Our group would like to recommend the following suggestions to help rebuild,
organize and streamline the CTLs help webpage in order to improve its accessibility
and usefulness.

After the examining the CTL tickets,

the evaluator team recommends that the following four topics be to add to the
CTLs eLC help webpage: I am having trouble logging in. Who can help?, How to
import quizzes, How to grade quizzes, How to release grade items & date
restrictions. These topics are worded to reflect the current style of the help pages
 EDIT7350E eLearning Evaluation & Assessment | Dr. ChanMin Kim | Spring 2017

student FAQs and Faculty Support subpages.

1) The addition of analytics to the eLC webpage would be beneficial in order to

gauge the usefulness of various help guides to eLC users. Analytics would
provide data showing the number of users, location of users, and peak times for
the use of the page.

2) The addition of a Frequently Asked Questions (FAQ) Section would be helpful in

order to troubleshoot common topics.

3) Topics that are suggested to add are:

a) How to import a quiz
b) How to upload a video
c) How to grade quizzes
d) Releasing grade items & date restrictions
e) I am having trouble logging in. Who can help?

4) Navigation of the eLC Support Page may be improved by implementing a

hierarchical approach to displaying help topics.

5) Creating an initial page inquiring about the users level of experience will allow for
filtering of the results received during searches.