Selection of Web-based e-Tools for Global e-Universities

1. Introduction

I believe that there is a growing feeling of disenchantment with two aspects of the software life cycle: the traditional process of specification of systems to be developed by software development companies (such companies may be cheap but often have no track record or experience of large systems) and the current adversarial procurement methods often used by government agencies (in the UK and EU).

Thus I propose that we look for new paradigms of procurement. The work was informed by (among other things) the handling of the SuperJANET4 procurement by JISC. The top-level observation I make is that a new paradigm of procurement does not depend in detail on the functionality of the system being procured. The two key features of my proposed new generation procurement model are: 1. procurement as a conversation between customer and supplier business models, which iterates towards best value as feature inclusion/exclusion is negotiated using cost and timescale factors 2. procurement based on generalised features rather than on (in some cases) over 100 specific features of the functionality or user interface. This model was first described by me in May 2000, in order to handle the analysis of MLE vendors in the e-tools (1) study for the HEFCE e-University Planning Team [Bacsich et al, 2000]. In that report I recommended concentration on just 12 features: 1. Architectural approach of the system including any restrictions on the content that it can deliver (e.g. mathematics, chemistry) 2. Standards and interoperability, including compliance with current and emerging standards fora, such as IMS and CEN/ISSS; and interoperability with other types of system including student records systems and learning management systems. 3. Life-cycle costs thus not just purchase cost but also recurrent costs of ongoing support, software upgrades, training, etc 4. Scalability up to the largest university scale of use (i.e. several hundred thousand users within a few years); and the footprint required to run the client system 5. User interface, including compatibility with Internet browsers and other major packages that students are likely to use for word processing and email; plus user interface issues for tutors, administrators and developers 6. Reference sites three reference sites of most relevance to the prospective customer, with ideally, at least one of these sites a university (in the relevant country) making substantial use of the system 7. Reliability of the system, both server and client, including any measurements of this 8. User empowerment, including full details of how students, tutors, administrators and others can customise the system 9. Company size and stability 10. Ease of support (and training), including details of how user sites will acquire their training (e.g. from vendor, independent trainer, self-training material, etc); and details of
Bacsich (WWW10 paper) 1 February 2001

Selection of Web-based e-Tools for Global e-Universities Towards a New Paradigm

what specialist training is needed (e.g. for tutors, administrators, course developers, systems developers) 11. Current and proposed capability to embed new technology, such as wireless, mobile and fibre which will change the parameters of many systems including allowing full-motion video to be an object anywhere in the system; and developments of non-PC devices such as palmtops and set-top boxes 12. Current and proposed capability to embed new pedagogy under development by educational researchers, such as Virtual Labs, knowledge management, performance support and co-operative knowledge building.

2.

Further work

In academic work over the Autumn 2000 period since the e-tools (1) study, I have been been refining this process model for procurements, with particular application to Managed Learning Environments, but susceptible to much wider use.

2.1.

Analytic work in Canada

In Canada, the well-regarded project team of Bruce Landon and co-workers have constructed a Web-based assessment tool to compare online educational delivery applications. Over 40 products are reviewed. The main criteria are similar to a subset of those in the e-tools (1) report, but there is also a mass of detail which is in our view not helpful to selection of e-tools to be deployed on a massive scale. Details are available at www.ctt.bc.ca/landonline/

2.2.

Analytic work in Australia

In Australia, the analytic work of Anne AHerran at James Cook University, Townsville, Australia, has been presented at various conferences and presentations in Australia and the UK. It is well documented on the Web at www.tld.jcu.edu.au/general/survey_re/ Lists of evaluation criteria tend to grow until they are unworkable, for example the early Design Requirements published by IMS (Instructional Management Systems) which listed 186 criteria. At the other end of the spectrum Betty Collis (Twente University, Netherlands) acknowledged when visiting JCU (September 1999) that her primary criterion was server independence. Shirley Alexander and James Sawers at UTS based their evaluation on eight essential criteria (A centralised approach to the adoption of a university-wide web-based learning tool Ascilite 1998). In a field where nothing sits still for long it is important not to place too much dependence on features which continually update. Their volatility renders invalid many evaluations too dependent on feature sets. System criteria versus Learning materials / course criteria A distinction is drawn at outset between features which relate to the quality of teaching and learning in online courses, and features which relate to the system as a vehicle. The pedagogical quality of learning materials is the responsibility of this institution through its teachers and course developers, (see Recommendation 5: Quality Standards) and is not a function of the system which delivers them.

Bacsich (WWW10 paper)

February 2001

Selection of Web-based e-Tools for Global e-Universities Towards a New Paradigm

selection criteria for an online system do not therefore include criteria that relate to the pedagogical quality of learning materials but only criteria that relate to the capacities of the system itself. Key selection criteria (generic) Our list originally identified approximately 100 criteria. These were reduced to the essential and the desirable. In the end 19 key criteria were given prominence. Previous analysis (OReilly & Walker 1998) identified 13 key attributes of a good system. User feedback at JCU and a review of the literature confirmed the relevance of these features. They evenly represent the needs of the four main stakeholders in an online system, with a fair amount of overlap. Apart from the overarching stakeholder: the institution, with its interest in marketing and extending its offerings online, these are: Administrators, Technicians, Course Developers or Teachers, and Learners. We intend to take a similar middle of the road approach in this paper, using a small number of additional institutional/pedagogical criteria to merge with the 12 system criteria of the e-tools (1) report. See the next section.

2.3.

Procurement criteria an update

This section updates the procurement criteria from the e-tools (1) report in the light of sectoral and other information. There were twelve criteria, as described in the subsections below. Five related criteria were given in the Hambrecht report [Urdan & Weggen, 2000]: Leveraging existing open industry standards Scalable to any size enterprise Flexible technology to include groups of workstations or the entire organisation Easy integration with clients internal systems Delivering media rich broadband experience For Administrators: Scalability, Value for money, Integration with existing systems For Technicians: Robustness, User base, Technical Support, Ease of maintenance For Course Developers or Teachers: Customisability, Flexibility, Integration of legacy materials For Learners: Consistency, Accessibility, Quality of design Typically software tools are evaluated on the basis of the features they provide, their technical specifications and cost. In this report we argue that additional pedagogic criteria are required in order to differentiate VLEs with respect to their use in different teaching and learning situations. We integrate all these criteria into one system, by adapting the e-tools (1) criteria, as follows: 2.3.0. Overall functional specification There still needs to be an overall specification of the system functions. However, instead of features being either mandatory or optional, it is more useful if a measure can be given of how essential they are (e.g. on a score of 1-10).
Bacsich (WWW10 paper) 3 February 2001

Thirteen related criteria were developed by AHerran in her Australian work:

The JISC JTAP report on VLEs claims:

Selection of Web-based e-Tools for Global e-Universities Towards a New Paradigm

2.3.1. Architectural approach This should include any restrictions on the content that the LMS can deliver (e.g. whether it can deliver mathematics or chemistry). This should be broadened to include Hambrechts criteria of Flexible technology to include groups of workstations or the entire organisation and Delivering media rich broadband experience. It should also be further broadened to include AHerrans Course Developers: Integration of legacy materials. 2.3.2. Standards and interoperability This should include compliance with current and emerging standards forums, such as IMS, plus interoperability with other types of system including student records systems. This already includes Hambrechts Leveraging existing open industry standards and Easy integration with clients internal systems, as well as AHerrans Administrators: Integration with existing systems. 2.3.3. Life-cycle costs This should cover cost indications, including not just purchase cost but also cost for ongoing support, software upgrades, training, etc. This criterion already includes AHerrans Administrators: Value for money. 2.3.4. Scalability (including footprint issues) This should include a description of how the system can grow to support the expected number of users within a few years. By footprint we mean the configuration required to run the client end of the system, and how this compares with similar systems. This criterion already includes Hambrechts Scalable to any size enterprise and AHerrans Administrators: Scalability. 2.3.5. User interface (including internal and external consistency) This should include compatibility with Internet browsers and other major packages that students are likely to use for word processing and email; plus user interface issues for tutors, administrators and developers. This should be broadened to include AHerrans Learners: Consistency, Accessibility, Quality of design. 2.3.6. Reference sites This should describe reference sites of most relevance to the institution. Ideally, at least one of these sites should be a UK university making substantial use of the system. Other sites may be universities or corporate universities in the UK or elsewhere, but vendors should ensure that the sites are as relevant as possible (in their judgement) to the UK context. This already includes AHerrans Technicians: User base. 2.3.7. Reliability How reliable is the system, both server and client? What measurements do vendors have? This already includes AHerrans Technicians: Robustness.

Bacsich (WWW10 paper)

February 2001

Selection of Web-based e-Tools for Global e-Universities Towards a New Paradigm

2.3.8. User empowerment This should include full details of how students, tutors, administrators and others can customise the system. For example, some systems are extremely easy to use, but extremely hard to customise. This already includes AHerrans Course Developers: Customisability and Course Developers: Flexibility. 2.3.9. Company size and stability This should include, if the company is wholly devoted to e-learning, please give details of company sales over the few years, and other evidence of stability. If the company has a Division devoted to e-learning, this should include figures for that Division. If the company has several products, this should include a breakdown of figures down to the product level. 2.3.10. Ease of support (and training) This should include full details including typically how user sites will acquire their training, e.g. from vendor, independent trainer, self-training material, zero training need; plus details of what specialist training is needed, e.g. for tutors, administrators, course developers, systems developers. This already includes AHerrans Technicians: Technical Support and Technicians: Ease of maintenance. 2.3.11. Current and proposed capability to embed new technology New forms of networking such as wireless, mobile and fibre are coming along which will change the parameters of many systems including allowing full-motion video to be an object anywhere in the system. There are also developments of non-PC devices such as palmtops and set-top boxes. Vendors should explain how the architecture of their system and structure of their company will allow them to adapt to such technologies. 2.3.12. Current and proposed capability to embed new pedagogy Educational researchers continue to develop new approaches to teaching, often exemplified by hard-to-deploy technology. Current hot topics include Virtual Labs and co-operative knowledge building. Vendors should explain how they make their system open to new pedagogic approaches. In the light of the JISC JTAP report, and input from the sector, this should be broadened to state that the system should have capability to handle all the current pedagogic techniques that are likely to be in cost-effective use within the target institution(s).

3.

Final comments and conclusions

Some similar thoughts, including support for the concept of gap analysis, can be found in the TMG Corporation report by Richards [2000]. This also provides backing for the idea of using an off-the-shelf (with modifications) approach as a middle way between the false dichotomy of off-the-shelf and custom development paradigms. Another interesting idea is the two-stage procurement process used for the eArmyU systems. Our conclusion is that our methodology has been refined rather than challenged by the information coming in over the last few months.

Bacsich (WWW10 paper)

February 2001

Selection of Web-based e-Tools for Global e-Universities Towards a New Paradigm

4.

References

1. AHerran, A. (2000). Report & Recommendations: Research and evaluation of online systems for teaching and learning, James Cool University, Australia, March 2000, http://www.tsd.jcu.edu.au/develop/survey_re/recs/recs1.html 2. Bacsich, P., Heath, A., Lefrere, P., Miller, P., Riley, K. (1999). The Standards Fora for Online Education, D-Lib Magazine vol.5 no. 12, December 1999, http://www.dlib.org/dlib/december99/12contents.html 3. Bacsich, P. (2000). E-University developments and issues in Britain, Presentation to the EuroPACE Wanderstudent conference, EuroPACE, Leuven, November 2000. 4. Bacsich, P., Lefrere, P., Mason, R. (2000). Survey of Pedagogic, Assessment & Tutoring Tools: e-tools for the e-University Survey 1, HEFCE, September 2000 (confidential). 5. Britain, S., Liber, O. (1999). A Framework for Pedagogical Evaluation of Virtual Learning Environments, University of Wales Bangor, JTAP Report 41, October 1999, http://www.jtap.ac.uk/reports/word/jtap-041.doc 6. Dean, Christopher, (2000). Technology Based Training & On-line Learning, DfEE, 2000. 7. JISC, Circular 7/00 Managed Learning Environments in Further Education: progress report, JISC, July 2000. 8. Landon, B. et al. (2000). Online educational delivery applications: a Web tool for comparative analysis, http://www.ctt.bc.ca/landonline/, last updated June 2000. 9. Richards, C. (2000). Selecting a Learning Management System you can live with, TMG Corporation, Vermont, 2000. 10. Urdan, T., Weggen, C. (2000) Corporate e-learning: exploring a new frontier, W R Hambrecht & Co Equity Research, March 2000.

Bacsich (WWW10 paper)

February 2001