Learning Management System (LMS) Review Committee

Final Report
May 5, 2014
Beth Rubin, Ph.D. Assistant Provost for eLearning
The Learning Management System (LMS) Review Committee was initiated in January of 2014,
with representatives from all Miami divisions. Members include faculty (Jason Abbitt, Mike
Brudzinski, Al Cady, Bruce DArcus, Michele Dickey, Michele Gingras, Lynette Hudiburgh, Norm
Krumpe, Tom Mayes, Mark McBride, Glenn Platt) and staff (Jason Cardoso, Kent Covert, Janet
Hurn, Pete Natale, Beth Rubin (chair), Lisa Santucci).
The committee established a public Google folder, including all documentation of the process
(https://drive.google.com/a/miamioh.edu/?
tab=co&urp=https://drive.google.com/a/miamioh.edu/?tab%3Dco
%26aut#folders/0B0DMdewEr4Yxa1BEajFfVEFXMjQ). Faculty members received copies of
evaluations of LMSs conducted by Indiana University. In its initial meeting on February 7, the
committee developed a process and finalized a set of criteria to use when evaluating LMS
systems. It identified working subcommittees to develop a Request for Proposals (RFP), as well
as a set of criteria reflecting essential characteristics to evaluate the proposals, a timeline, set of
vendors that should be sent the RFP; and a set of responses to Frequently Asked Questions.
The subcommittee drafted an RFP, and received nine responses. The set of essential criteria
were turned into an evaluative framework, used to review all proposals. All vendors with 0 or1
from at least two evaluators on any one item were eliminated from consideration, and the
numerical assessments were also aggregated. The result was two vendors that passed the
evaluation: Desire2Learn (D2L) and Instructure/Canvas.
These vendors presented to the university committee on March 17 at the Shriver Center, and
projected via WebEx to the university committee; they were also recorded and the recordings
placed on the public Google Site. Presentations were publicized via the Miami eReport and the
Chairs Email List. Evaluations were distributed to all observers; this survey reflected the full set
of criteria identified to evaluate LMSs. Approximately 50 people observed each presentation,
with around half watching on WebEx. Twenty-three evaluation forms were completed; Canvas
had higher average evaluations on 9 items, D2L on 5 items, although some differences were
extremely small (e.g., less than 0.1 on a 1-4 scale).
Following the presentations, all faculty and staff were invited to participate in the evaluation of
the two LMSs as well as Niihka. A short course was created on each system, and participants
were asked to conduct the tasks that they would when teaching, developing or taking a course
supported by an LMS (e.g., uploading a syllabus, giving formative feedback). Each task was
then assessed using a survey. Invitations were distributed using the Provosts All Faculty email
list and the ALT website; in addition, post cards were sent on April 8 to all full and part-time
faculty at all campuses. Committee members were asked to invite their colleagues and students
to participate, and a sample slide deck was prepared to facilitate presentations. In addition, a

LMS Review Committee 5/20/14

set of 21 testing lab sessions were held across Miami campuses, with staff available to support
faculty, staff and student testing. A raffle was offered to provide further encouragement for
faculty to participate, with one entry for every evaluation submitted. Testing continued for
approximately one month, from March 20 through April 21. During this time, two committee
members who are power users of Niihka responded to the vendor presentations, providing
context for the committee by comparing the tools of the new LMSes to their experience of using
Niihka.

Twenty-four faculty or staff evaluated D2L, 20 evaluated Canvas, and 23 evaluated Niihka; both
basic and advanced tasks were evaluated. In addition, three students evaluated the LMSs. Data
were aggregated and analyzed into a series of reports:
Report of Vendor Presentations Surveys
Report of Faculty Testing Surveys
Report of Faculty Test Comparisons on Specific Tasks
Report of Student Testing Surveys

Other reports were also produced, including:

LMS IT Considerations
LMS Assessment Report
LMS Migration Report
A member of the Miami Accessibility Committee conducted an evaluation of accessibility of the
potential new LMSs to people with disabilities, and drafted the LMS Accesibility 04282014
report. This evaluation found that both systems had processes to ensure accessibility, and both
systems were generally accessible to people with visual impairments although each has a
subsystem that is not yet fully accessible. D2L conducts an annual review to match its annual
release cycle. Canvas, after initially having strong accessibility, had developed limitations due to
system changes. However, Canvas has remediated the problems and implemented an ongoing
review to match its continuous release cycle, including hiring an employee analyst who employs
a screen reader. The Miami Accessibility Committee member joined that committees
communication list.
In addition, the committee reviewed Indiana Universitys LMS evaluation final report, which was
summarized by a committee member. It also reviewed D2Ls pricing proposal, which had been
revised a week prior, as well as Canvass pricing proposal. The committee calculated that both
systems were quite similar in pricing for the same set of options, although D2L had additional
optional packages; while D2Ls first year proposal was slightly lower, its cost increased annually,
and the average annual cost of Canvas over 5 years was slightly lower.

o
o

The committee held a two-hour meeting on May 5, and considered all data and reports. After
reviewing all evidence, the committee will voted on the following recommendations:
Miami University should change from Niihka to a new LMS. The result was:
Yes: 12
Abstain: 1
Miami University should adopt D2L or Canvas. The result was:

LMS Review Committee 5/20/14

o
o

Canvas: 12
D2L: 1
As a result, the committee recommended to the Provost that Miami University should adopt
Canvas as its LMS.
As of May 2014, a review of Canvas is being conducted by General Counsels office, and
financial implications are being assessed by means of a LEAN analysis. An implementation
team will be formed as soon as the acquisition is confirmed; this team will oversee the transition.
Academic representation will be sought from CoAD. Purchasing will be asked to negotiate a
price, and the new system should be initially set up by the end of June, 2014. Instructional
Designers will design course formats, and new courses developed in the summer term will be
developed on the new LMS. Course materials will be transferred from Niihka to the new system,
although automated transitions generally do not produce usable classes as materials need to be
organized and uploaded into various system tools.
Some faculty teaching online in fall 2014 will be invited to shift to the new LMS, and a phased
rollout will be implemented. Trainings will begin in summer of 2014 and continue through the
year, and student workers will be employed to re-create courses for faculty who choose to pilot
the new system, term by term. Niihka will continue to be available to all faculty and staff through
summer of 2015.

LMS Review Committee 5/20/14