The current issue and full text archive of this journal is available at www.emeraldinsight.com/0309-0590.

htm

PERSPECTIVE ON PRACTICE

Training evaluation: an analysis of the stakeholders evaluation needs

Marco Guerci
Department of Management, Economics and Industrial Engineering, Politecnico di Milano, Milan, Italy, and

Training evaluation

385
Received 2 March 2010 Revised 10 June 2010 Accepted 19 July 2010

Marco Vinante
Milan, Italy
Abstract
Purpose In recent years, the literature on program evaluation has examined multi-stakeholder evaluation, but training evaluation models and practices have not generally taken this problem into account. The aim of this paper is to ll this gap. Design/methodology/approach This study identies intersections between methodologies and approaches of participatory evaluation, and techniques and evaluation tools typically used for training. The study focuses on understanding the evaluation needs of the stakeholder groups typically involved in training programs. A training program nanced by the European Social Fund in Italy is studied, using both qualitative and quantitative methodologies (in-depth interviews and survey research). Findings The ndings are as follows: rst, identication of evaluation dimensions not taken into account in the return on investment training evaluation model of training evaluation, but which are important for satisfying stakeholders evaluation needs; second, identication of convergences/divergences between stakeholder groups evaluation needs; and third, identication of latent variables and convergences/divergences in the attribution of importance to them among stakeholders groups. Research limitations/implications The main limitations of the research are the following: rst, the analysis was based on a single training program; second, the study focused only on the pre-conditions for designing a stakeholder-based evaluation plan; and third, the analysis considered the attribution of importance by the stakeholders without considering the development of consistent and reliable indicators. Practical implications These results suggest that different stakeholder groups have different evaluation needs and, in operational terms are aware of the convergence and divergence between those needs. Originality/value The results of the research are useful in identifying: rst, the evaluation elements that all stakeholder groups consider important; second, evaluation elements considered important by one or more stakeholder groups, but not by all of them; and third, latent variables which orient stakeholders groups in training evaluation. Keywords Training evaluation, Stakeholder analysis Paper type Research paper

1. Introduction The market that is, buying skills and services from training providers was once the approach used by company training systems to establish relationships with groups
The authors are grateful to Dr Brian Bloch for his comprehensive editing of the manuscript.

Journal of European Industrial Training Vol. 35 No. 4, 2011 pp. 385-410 q Emerald Group Publishing Limited 0309-0590 DOI 10.1108/03090591111128342

JEIT 35,4

386

outside the rm. Presently, however, companies also try to establish such relationships through participation in public programs typically nanced by public bodies and intended to encourage and stimulate continuous training, which is considered to be a collective good. In training processes delivered in such contexts, many actors are required to make decisions which may have an impact on the performance of the training initiative. Typically, these stakeholders have different institutional missions, and their training interests and objectives may be different as well: their inclusion in the evaluation process creates and maintains diversity within the participating stakeholder group (Wills, 1993; Mathie and Greene, 1997). Furthermore, stakeholders can be particularly helpful when reviewing evaluators recommendations for program revisions. Recommendations to program personnel are commonly expected in evaluation reports (Brandon, 1999, p. 363). This study focuses on training evaluation in such multi-stakeholder contexts, and the aim is to identify intersections between two different disciplines. The rst is program evaluation, a formalized approach to the study of the goals, processes, and impacts of projects, policies and programs implemented in public and private sectors. The second is training and development management, and in particular, the literature on the evaluation models and tools used to evaluate training within companies. This study focuses on a continuous training project nanced by an Italian public authority, in order to highlight the evaluation needs of the stakeholder groups typically involved in this kind of training process. In the research reported here, stakeholders were contacted after the planning and delivery of the programs training modules, using qualitative and quantitative methods. In particular, the research process consisted of an initial qualitative research phase of key informants belonging to the different stakeholder groups, followed by quantitative research on the entire population. The results of the research can be applied usefully to various different purposes. First, the study identies convergences and divergences between the evaluation needs of the different stakeholder groups. Second, it identies the guidelines which orient stakeholder groups in training evaluation. 2. Literature analysis The literature analysis focuses on three issues. First, the theory on training evaluation is analyzed, indicating the theoretical reasons why the stakeholder-based evaluation applied to training can be considered important. The ndings in the literature on training stakeholder-based evaluation are then presented. The nal part of the survey concerns the contexts in which company training systems operate, and demonstrates the practical importance of stakeholder-based training evaluation. Overall, this section highlights knowledge gaps and denes specic research questions. 2.1 The theoretical background to training evaluation: why a stakeholder-based training evaluation? Training and education are an investment from which the organisation expects a positive return; that is, a return on investment (ROI) from training and education. For this reason, starting from the hierarchical evaluation model of Kirkpatrick (1998) and Phillips (1996) proposes a ROI training evaluation model which comprises ve levels; each investigating different elements:

. .

Level 1. Reactions: measures programme participant satisfaction. Level 2. Learning: focuses on what participants have learned during the programme. Level 3. Application and implementation: determines whether participants apply what they learned on the job. Level 4. Business impact: focuses on the actual results achieved by the programme participants, as they successfully apply what they have learned. Level 5. ROI: compares the monetary benets from the programme with the programmes costs.

Training evaluation

387

This model has made valuable contributions to training evaluation theory and practice, because it stresses the importance of thinking about and assessing training within a business perspective. Nevertheless, the model has at least three limitations. First, the model concentrates on a restricted set of variables. In fact, the ve levels of evaluation, which it proposes are based on an extremely simplied view of training effectiveness. In particular, they do not consider a wide range of organisational, individual, training-design and delivery factors that may inuence training effectiveness (Wills, 1993; Bramley and Kitson, 1994; Cannon-Bowers et al., 1995; Ford and Kraiger, 1995; Salas and Cannon-Bowers, 2001; Tannenbaum and Yukl, 1992; Kontoghiorghes, 2001). The second criticism concerns the causal linkages among training outcomes at different levels. That is, it is not possible to achieve positive results at top levels, if this did not occur at the lower levels as well. Research (Alliger and Janak, 1989; Talbot, 1992; Alliger et al., 1997) in the eld has largely failed to conrm such causal linkages. A third weakness of the hierarchical model of evaluation is that it lacks a multi-actor perspective. In fact, the point of view assumed by the model is that of the companys shareholders. Indeed, the model assumes that each level of evaluation provides data that is more informative than the last (Alliger and Janak, 1989). This assumption has generated the perception among training evaluators that establishing level four results will provide the most useful information about training program effectiveness (Bates, 2004, p. 342). As a consequence, the evaluation needs of the stakeholders involved in the training process are neglected, and this is particularly restrictive in contexts characterized by the presence of a plurality of actors. Applying stakeholder-based evaluation to training may be useful in dealing with this nal criticism by including the different points of view of the stakeholder groups in the evaluation programs design and implementation (Bramley and Kitson, 1994; Mathie and Greene, 1997; Mark et al., 2000; Holte-McKenzie et al., 2006). This could also impact on the rst criticism, because designing the evaluation program on the basis of stakeholder evaluation needs entails extending the set of variables considered by the ROI training evaluation model. 2.2 Studies on stakeholder-based training evaluation For some years, the literature on program evaluation has dealt with the topic of multi-stakeholder evaluation (Gregory, 2000; Mark et al., 2000), although reection on the issue and practical evaluation in the training eld have been less evident (Lewis, 1996). In fact, the best-known model of training evaluation is based almost exclusively on measuring results from the perspective of one single actor. This actor corresponds

JEIT 35,4

largely to the companys shareholders, considered as the subjects that fund training programs. This inevitably induces the evaluation system to focus on the impact, in nancial or operational terms, of training on company performance, without considering the effects on other stakeholders:
Stakeholder-based evaluation is an approach that identies, and is informed by, particular individuals or groups. Stakeholders are the distinct groups interested in the results of an evaluation, either because they are directly affected by (or involved in) program activities, or because they must make a decision about the program or about a similar program (Michalski and Cousins, 2000, p. 213).

388

The literature on stakeholder-based evaluation states that if evaluation is to improve program performance, it has an instrumental use and must be structured as a system, which supports actions, and even more so, decision-making processes (Flynn, 1992). For this reason, it is necessary to know the evaluation needs of the actors involved in the program whose evaluation system has to be designed:
Instrumental use, perhaps the earliest type of use examined in literature, refers to using evaluation ndings as a basis for action [. . .] Examples of instrumental use include eliminating a program shown to be ineffective, modifying a program based on an evaluation, targeting a program to new audiences, allocating new budget outlay for a program and changing the structure of the organization in which a program operates (Burke Johnson, 1998, p. 94).

It is consequently important to activate a stakeholder-based evaluation process that involves the actors. According to the theory on participatory evaluation (Cousins and Whitmore, 1998; Michalski and Cousins, 2000), such inclusion can be practical, when its purpose is to improve the programs performance or transformative, when it aims to emancipate the disadvantaged social/cultural groups at which the program is targeted. This classication is consistent with the more general theories of stakeholder management, which are: . instrumental theory of stakeholder management, grounded on the assumption that organisations which establish relationships with stakeholders, based on trust and collaboration will have competitive advantages compared with companies which do not establish such relationships. The competitive advantages derive from the fact that relationships based on mutual trust and cooperation facilitate efcient agreements which minimize transaction costs (Friedman and Miles, 2006); and . ethical-normative theory of stakeholder management, which argues that the normative base of the theory, including the identication of moral or philosophical guidelines for the operation and management of the corporation is the core of the stakeholder theory (Donaldson and Preston, 1995, p. 71). Based on such considerations, various studies have discussed the topic of stakeholder-based training evaluation by adopting the concept of practical participatory evaluation, which itself is based on the more general, instrumental theory of the stakeholder management. Hence, this research strand denes a stakeholder as a subject able to inuence the performance of a training process, because she/he is requested to make decisions during the process. It also conceives the evaluation system as an instrument for providing the stakeholders with the information necessary to validate the decisions they are requested to make.

The studies, which delve further into stakeholder-based evaluations applied to training, divided into two broad categories. Studies in the rst category provide a theoretical view of the topic and dene which evaluation process should be used for stakeholder-based training evaluation (Reineke, 1991; Talbot, 1992; Wills, 1993; Brandon, 1998; Brandon, 1999; Bates, 2004; Nickols, 2005; Shridaran et al., 2006). These studies adopt the instrumental theory of stakeholder management and consider the participation in evaluation as practical. In fact, the assumption is that, in order to maximize the return on training investment, there has to be a balance between the contributions the training process receives from stakeholders and the incentives that they receive in return (Nickols, 2005). For instance, managers that nance the training program invest resources, in order to exert a positive impact on the organisations business performance or on the individual performance of participants. The trainees, in turn, participate with their efforts, attention and time in the hope of acquiring new knowledge and learning concepts, methods, and tools that are useful for their careers. The various stakeholder groups must perceive a value in this exchange: that is, the incentives must have a value equal to, or greater than, the contributions. The evaluation plan therefore enables the stakeholder groups involved in the program to monitor the added value of this exchange. On the basis of the above-mentioned approach, this research strand denes the evaluation process to be implemented for a training stakeholder-based training evaluation (Figure 1). The second category of studies concerning stakeholder-based training evaluation analyzes the evaluation needs of the stakeholder groups typically involved in a training program. That is, they deal with the elements of evaluation they consider useful for monitoring the balance between contributions and incentives. Such studies have demonstrated the existence of signicant differences between the evaluation needs of stakeholder groups. In particular, they have focused on: . the evaluation needs of the stakeholder groups within the company, that is, managers, training experts and participants (Brown, 1994; Michalski and Cousins, 2000); and . the evaluation needs of the stakeholder groups outside the company, that is, external training providers, public training schools and trade unions (Garavan, 1995). 2.3 How company training systems operate: the importance of stakeholder-based training evaluation Many companies have established training systems dedicated expressly to providing the training support necessary to implement corporate strategy. Such systems interact constantly with the external environment, with which they exchange practices, resources and competencies. In particular, an analysis of the relationship frames that these training systems form with actors outside the company shows that there are two different options that companies may pursue. The rst option entails the use of the companys resources to purchase services directly from external training providers. The second option consists of taking part in public policy programs typically nanced by third parties. These programs have the specic purpose of producing collective goods (Ostrom, 1990), such as an increase in the employment rate, a greater competitiveness of small and medium-sized businesses, and innovation[1].

Training evaluation

389

JEIT 35,4

Macro-phase 1: Evaluation program design Identification of the significant stakeholder classes within the training program

390

Identification, per class, of the program performances able to provide the expected benefits and (where necessary) identify a scale of priorities per class and per performance

Definition (where necessary) of a weight for each stakeholder class in relation to its capacity to influence the program's performance

Design of an evaluation program for the performances identified for each class, and design of data reporting

Macro-phase 2: Evaluation program implementation Collection, processing and data presentation to the stakeholder classes according to the above-defined times and conditions

Figure 1. Stakeholder-based evaluation process

Periodic assessment of the stakeholders' satisfaction with the evaluation system Source: Adapted from Nickols (2005)

On selecting one or the other option, a company chooses different regulation systems distinguished by different principles and rules used for the allocation of resources among actors (Polanyi, 1944). The public-policy regulation system has the following basic characteristics (Meny and Thoenig, 1989): it is a response to collective demands and requirements; it is extremely complex from both decisional and implementation perspectives; its purpose is to encourage changes in specic populations; and it uses ad hoc instruments and procedures, combined with incentives to achieve the desired behaviour. In terms of resources allocation, the most important characteristic of this regulation system is the presence of a public authority that denes the principles of resources allocation in different areas (for instance training, work, health, etc.) and in regard to different subjects (individuals, families, workers, companies, etc.). The other regulation system is the market. This is based on the interaction between training demand and supply. The operating model for this system, therefore, requires no regulation processes governed by any third-party but is determined mainly by prices, which act as self-regulating mechanisms (Polanyi, 1944). What is relevant for the purposes of this paper is the fact that the public-policy regulation system may be considered a privileged context for the application of stakeholder-based training evaluation because:

a number of actors (stakeholders) are requested to make decisions that have an impact on the programs performance; and these actors may have institutional missions, and consequently interests, objectives and evaluation needs which may not be entirely convergent.

Training evaluation

3. Knowledge gaps and research questions The above survey of the literature has shown that a stakeholder-based approach to training evaluation is useful for two reasons. First, it enables the design of evaluation programs, which the actors involved in the training program can actually use to support their decisional processes. Second, it expands the range of the variables considered by the ROI training evaluation model, normally focused on the evaluation needs of the companys shareholders (Phillips, 1996; Kirkpatrick, 1998; Ross, 2008). These two advantages are signicant because: (1) company training systems are structured as open systems which participate in networks outside the company, and which involve a number of actors that make decisions with an impact on training program performance; and (2) such networks are often part of the public-policy regulation system governed and nanced by a third-party. Consequently, they include actors with different institutional missions, and, therefore, specic evaluation needs. This study belongs to the second strand of research on stakeholder-based training evaluation, because its aim is to identify the evaluation needs of the stakeholder groups typically involved in a training project within the public-policy regulation system. The results of the study are useful in identifying: . convergences, that is, the evaluation elements that all stakeholder groups consider important; . divergences, that is, evaluation elements considered important by one or more stakeholder groups, but not by all of them; and . latent variables, identied through a factor analysis, which orient stakeholder groups in training evaluation and convergence/divergences among stakeholder groups in the attribution of importance to these variables. 4. Methodology and research process The research reported by this study selected a training program nanced by the Italian Lombardy Region (European Social Fund, D1). The program focused on the promotion of a competent, qualied and adaptable workforce, and its objective was to implement training interventions which enhance the competitiveness of local manufacturing with particular reference to small and medium-sized businesses. Enterprise associations could submit their training proposals to the public authority, which then selected the relevant training providers. This case was chosen, because it was being implemented by means of cooperation and co-planning among enterprise associations, training providers and companies. The training program was, therefore, considered to be a privileged application of stakeholder-based training evaluation. The research process was divided into two phases: the rst was based on qualitative techniques, the second on the survey research method.

391

JEIT 35,4

392

4.1 First phase The rst research phase consisted of in-depth interviews, intended to identify the evaluation dimensions important for the various stakeholder groups involved in the training program. First, subjects able to supply information useful for the exploratory purposes of this phase were selected from each stakeholders group. The interviewee selection process identied key informants able to provide items helpful for reconstructing the evaluation needs of each stakeholder group. In light of the theory on stakeholder management in evaluation programs (Rossi et al., 1999), the involvement of the following stakeholder groups was considered essential: . Target participants. those at which the training program is aimed. . Decision makers. the actors who activated and nanced the training program; they were also responsible for monitoring it. . Program staff. the actors who carried out or supported the activities included in the program. . Program managers. the actors who supervised and managed the program. . Contextual stakeholders. the actors operating in the environment surrounding the program and who also had to make decisions which might inuence the results. Table I shows the actors interviewed for each stakeholders group. The interview structure can be illustrated by referring to the concepts of principle, dimension (or, in the case of the evaluation research result dimension) and indicator. Figure 2 shows the logical-formal relationships among these concepts. A principle is a general viewpoint which helps to orientate the evaluation to dened areas. Examples of principles are the effectiveness of the program, its efciency, fairness, and so on. A dimension is the rst breakdown level of a principle: while a principle is, by nature, general and partly uncontextualised, dimensions are more
Number of subject interviewed 3 1 3 2 4 12

Stakeholder groups Target participants Decision makers Program staff

Actors identied within the training program

Table I. Stakeholder groups, actors identied, subjects interviewed

Training participants Manager of the public body nancing the program Trainers on the program conducted by the training providers Program managers Program manager and project coordinators of the program by the training providers Contextual stakeholders Training managers of the companies involved in the program; manager of the enterprise association responsible for training activities

Principle

Figure 2. Principles-dimensionsindicators

Dimension Indicator Indicator Indicator

Dimension Indicator

specic and concern the object of evaluation. These dimensions can also be related to result dimensions, which are the specic, actual results pursued by the program. Finally, the indicator is a tool used to classify, categorize, and/or measure a dimension (Lazarsfeld and Rosenberg, 1955; Scriven, 1993). According to the objective of this phase, the most suitable principles provided by the literature (Rossi et al., 1999) were selected and adapted. This yielded seven principles, as follows: (1) Efcacy. whether the training intervention is able to achieve its aims consistently with the needs expressed by the actors involved. (2) Efciency. the results compared to the resources invested. (3) Accessibility. whether the training initiative discriminates against certain groups in gaining its benets. (4) Image. positive effects on the image (internal and/or external) due to the organizations realization of/participation in the training program. (5) Multiplier/transferability effect. the interventions capacity to generate positive effects; more specically, reproducibility, or transferability, indicates whether an intervention can be repeated/used in other, similar contexts. (6) Innovation. the programs ability to diffuse previously unused practices within its context. (7) Synergy. the programs ability to maximize its results by interacting in coordination with other similar programs. Using these general principles adapted from the literature, it was hypothesized that each stakeholder group conceived the principles in a specic way and translated them into different result dimensions. Hence, each interview included the following questions relative to each principle: . From your point of view, with reference to the training program being evaluated, is this principle useful (for the evaluation)? . If yes, what dimensions would you use to assess this principle? Starting from the seven principles explained above, 35 result dimensions were identied; some of them were common to two or more stakeholder groups, others were specic to just one stakeholder group. 4.2 Second phase In order to determine the evaluation needs of the stakeholder groups in more detail, the second phase of the research examined the importance attributed by each stakeholder group to the result dimensions identied in the previous phase. A survey was carried out, using a structured questionnaire consisting of the following question for each result dimension (35 items): how important is it for you? (measured on a cardinal scale from 1 to 10). Before the questionnaire was administered extensively, it was tested by two subjects from each stakeholder group (in total, 12 subjects): this pre-test assessed comprehension of the questionnaire, the time required to complete it, the functional and discrimination capacity of the measurement scale.

Training evaluation

393

JEIT 35,4

394

Once the tool had been created and the pre-test phase completed, the questionnaire was sent by e-mail to all subjects belonging to the stakeholder groups involved in the training program. The population was composed as follows: 146 participants, 48 companies, 26 trainers, 16 training providers, three representatives of the company association, two representatives of the public authority nancing the training program. After the questionnaires had been dispatched, procedures were established to send reminders, using a personalized recall system, rst by e-mail, subsequently by phone. At the end of the eld phase, the subjects who had returned correctly completed questionnaires were the following: 38 participants, 21 companies, 22 trainers, nine training providers, three representatives of the company association, two representatives of the public authority nancing the training program. The response rate for each stakeholder group was different, and the rate per group is summarised in Table II. 5. Findings The main ndings can be summarised under the following headings: (1) the levels of importance attributed in total to the items by the sample of stakeholders; (2) differences among the different stakeholders groups regarding the importance attributed to the items; (3) latent variables underlying the 35 items; and (4) differences among the stakeholder groups regarding the importance attributed to the latent variables. These ndings are presented in greater detail below. 5.1 Levels of importance attributed in total by the sample of stakeholders to items Considering the items achieving the highest values on the scales (Table III[2]), four refer to the ROI training evaluation model (Phillips, 1996): Item 1 (satisfaction with didactics/training methods), Item 3 (quality and amount of knowledge and skills acquired by the participants), Item 10 (utility of acquired knowledge and skills for the participants) and Item 11 (satisfaction level of companies with the training program). The additional items, which integrate the ROI training evaluation model, essentially refer to:
Stakeholder groups Participants Companies Trainers Training providers Representatives of the enterprise association Representatives of the public authority Notes: The total number of responders amounted to 95 (a 39.4 per cent response rate) Response rate (%) 26.03 43.75 84.61 56.25 100.00 100.00

Table II. Response rates

Dimension 1. Satisfaction with didactics/training methods 2. Increase in training investment by companies 3. Quality and amount of knowledge and skills acquired by participants 4. Increase in public resources to be invested in continuous vocational training 5. More knowledge by training providers about company training needs 6. Utility of acquired knowledge and skills for the careers of participants 7. Alignment between level of training program and level (of knowledge/skill) of participants 8. Efcient use of the resources by training providers 9. Creation of a network among training providers, nancer and companies 10. Utility of acquired knowledge and skills for participants and their work on a short-term basis 11. Satisfaction level of companies purchasing the training program 12. Accessibility to training program (for workers/employees)

Mean 8.80 8.64 8.44 8.29 8.28 8.21 8.09 8.06 8.05 7.89 7.88 7.84

Training evaluation

395
Table III. Result dimensions considered most important by all stakeholder groups

. .

the training resources (efcient use of resources by training providers, increase in public resources to be invested in continuous training, increased investment in training by companies); access to training opportunities (accessibility to training program for workers); consistency between the training supplied and the requirements of companies and participants (more knowledge by training providers about company training needs, value of knowledge and skills acquired for the careers of participants, alignment between the programs training level and level of participants); and alignment between training demand and supply (creation of a network among training providers, nancer and companies).

The other dimensions not included in the ROI training evaluation model are considered important by all the stakeholder groups. Consequently, they must be considered in the evaluation design, if the evaluation plan is to satisfy stakeholder evaluation needs. 5.2 Differences among stakeholders groups regarding the importance attributed to the results dimensions This section analyses the attribution of importance by stakeholder groups: the differences highlighted, refer to the attribution of importance for subpopulations corresponding to the stakeholder groups dened in Table I. The statistically signicant differences among groups are: (1) Satisfaction level of companies with the training program, which is more important for the enterprise association and the public authority than for companies. (2) Possibility to dene training nancing procedures with the public authorities, which is important for the training providers (delegated to manage resources and account for them) and the enterprise association. (3) Improvement in the training providers image among companies, which is important for training providers and trainers, but less important for participants.

JEIT 35,4

396

(4) Transparency of the mechanism controlling access to nanced training services, which is important for the enterprise association, the training providers and the public authority, but considered less important for participants and trainers probably, who are probably more focused on micro-dimensions related to interaction processes in training settings. (5) Number of bureaucratic procedures imposed on participating companies, which is important for the enterprise association, the public authority, training providers and companies, whilst it is less important for participants and least of all for trainers; the latter, as for the previous item, seem more interested in the micro-dimensions associated with training processes. (6) Impact of the training program on company results, which is more important for the public authority, the enterprise association and companies, and less important for participants and for the training supply system (trainers and training providers). (7) Quality and level of knowledge and skills acquired by participants, which is more important for trainers and training providers, even more so than for participants. procedure)[3] was useful to rene the Analysis of the post hoc variance (Scheffe analyses, because it highlighted for six of the seven dimensions listed above which stakeholder groups were differentiated in attribution of importance. The following list presents these dimensions and the different attributions[4]: . The quality and level of knowledge and skills acquired by participants is more important for trainers than for participants. . The number of bureaucratic procedures imposed on participating companies is more important for training providers than for trainers. . The impact of the training program on company results is more important for companies than for trainers, but is important even for participants. . The transparency of the mechanism controlling access to nanced training services is more important for training providers and companies than for participants. . The improvement in the training providers image among companies is more important for training providers and trainers than for participants. . The possibility to dene training nancing procedures with the public authorities is more important for training providers than for trainers and participants. Table IV shows the differences among stakeholder groups with regard to the importance attributed to dimensions. 5.3 Latent variables underlying the dimensions The aim of the factor analysis was to extract, starting from the 35 dimensions, latent macro-variables representing a linear combination of the original variables and which were independent of each other[5]. The exploratory factor analysis was conducted on 26 items[6] and yielded ve factors explaining 65.5 per cent of the total variance (Appendix 3).

Training of trainers Training Enterprise Public Participants providers Trainers association authority Companies (TOT) * Satisfaction level of companies with training program Impact of training program on company results Possibility to dene the nancing procedures for training with public authorities Improvement in training providers image among companies Transparency of mechanism controlling access to nanced training services Number of bureaucratic procedures imposed on participating companies Quality and amount of knowledge and skills acquired by participants

Training evaluation

6.60 6.78 * *

8.66 8.55

8.50 7.95 * *

9.66 8.33

10 10

7.57 8.33 * *

7.88 7.68

397

7.00 * *

9.66 * *

7.00 * *

9.66

7.5

8.33

7.64

6.60 * *

9.11 * *

8.59 * *

7.33

7.5

7.28

7.49

6.68 * *

9.11 * *

6.81

9.33

8.47 * *

7.47

7.26

8.88 * *

6.09 * *

9.66

8.28

7.22

7.63 * *

8.88

9.36 * *

8.66

8.5

8.71

8.44

Notes: *The analysis of variance (ANOVA) one-way test shows that the dimensions listed in the chart have signicant differences between means for groups of respondents (sig. 0.01); * *ANOVA test, post , mean differences of statistically signicant groups (sig. 0.05) hoc by Scheffe

Table IV. Differences among stakeholder groups regarding the importance attributed to dimensions

The factor analysis obtained the following latent variables: (1) Support for the competitiveness of companies and human resources (Factor 1, Cronbachs alpha 0.866): training is considered (and evaluated) as a means available to companies and workers to improve performance and enhance the competitiveness of the economic and productive system. This factor refers to both company competitiveness and professional worker/employee development. (2) Promotion of fairness and image (Factor 2, Cronbachs alpha 0.805): training is considered (and evaluated) as promoting social equity training must also be made accessible to disadvantaged subjects in the system, including workers and companies and consolidating the image of actors in the external environment.

JEIT 35,4

398

(3) Network stabilization (Factor 3, Cronbachs alpha 0.844): training is considered (and evaluated) as a stable system of relationships that are nanced by public and private resources, managed jointly by the actors, and programmed for knowledge transfer. (4) Training services offer (Factor 4, Cronbachs alpha 0.807): training is considered (and evaluated) a provision of services for knowledge transfer; this supply chain must be efcient in order to reduce the costs to companies of accessing it and effective in order to achieve the training objectives. (5) Learner care (Factor 5, Cronbachs alpha 0.487[7]): training is considered (and evaluated) as a service which, therefore, considers mainly the variability and individual specicities within the training process. 5.4 Differences among stakeholder groups regarding the importance attributed to the latent variables The variables making up each of the ve above-described factors were used to create the respective measurement scales (indexes). The indexes (Table V) which totalled the highest attributions of importance were learner care (8.20) and network stabilization (8.18), while the index with the lowest attribution was promotion of fairness and image (7.33). The analysis of variance (ANOVA) one-way highlighted which indexes had signicant mean differences between stakeholder groups: they were competitiveness for companies and human resources, network stabilization, training services offer . The learner care index lay just above the acceptable limit of signicance, whilst the promotion of fairness and image index did not have an adequate signicance value (sig. . 0.05). procedure) showed that, for the index competitiveness of The post hoc test (Scheffe companies and human resources, two stakeholder groups were signicantly reciprocally differentiated in the attribution of importance; in particular, trainers and companies considered this factor more important than participants. Table V shows the differences among stakeholder groups in the importance attributed to latent variables. 6. Conclusions As reported in the literature review, the studies on stakeholder-based evaluations applied to training, divide into two broad categories: those that provide a theoretical view of the topic and dene which evaluation process should be used for stakeholder-based training evaluation; studies that analyze the evaluation needs of the stakeholder groups typically involved in a training program that is, the elements of evaluation that they consider useful for monitoring the balance between contributions and incentives. This paper is in the second category, as it identies the evaluation needs of the stakeholder groups typically involved in a training project. The research focused on an intervention in a public-policy regulation system (governed and nanced by a third authority), because it includes actors with different institutional missions, and therefore, specic evaluation needs. The outputs of this research are as follows: identication of evaluation dimensions not taken into account by the ROI training evaluation model of training evaluation, but important for satisfying stakeholder evaluation needs (LeBaron Wallace, 2008); identication of convergences/divergences among stakeholder group evaluation needs; identication of latent variables and convergences/divergences in the attribution of importance to them among, stakeholders groups (Michalski and Cousins, 2000).

Trainers 8.45 * 8.44 8.02 9.24 8.16 8.63 8.33 8.50 9.20 8.33 7.61 8.46 7.89 8.38 8.06 9.13 8.00 7.89 9.43 8.14 7.27 7.96 6.91 8.48 8.22 * 7.24 * 6.93 7.82 7.15 7.84

Training providers Companies

Public authority

Enterprise association

Participants TOT 7.93 7.33 8.18 7.42 8.20

Anova one-way (sig , 0.05) 0.000 0.128 0.019 0.023 0.053

1. Competitiveness of companies and HR 2. Promotion of fairness and image 3. Network stabilization 4. Training-services offer 5. Learner care

post hoc test, statistically signicant mean group differences (sig. 0.05) Note: *ANOVA test, Scheffe

Training evaluation

Table V. Differences among stakeholders groups regarding the importance attributed to latent variables

399

JEIT 35,4

400

Considering the results of the studies on stakeholder-based evaluation included in the second category (studies about the evaluation process to be used in participatory evaluation, and in particular, Nickols, 2005), the results of this paper are useful for designing before the training delivery an evaluation system for training programs. In particular, the results might be useful in the following phases of the stakeholder-based evaluation process: . Identication of the signicant stakeholder groups within the training program, as the paper suggests the stakeholder groups that are part of the decision-making process for a training program included in a public-policy regulation system. . Identication, per group, of the program performances able to provide the expected benets and (where necessary) identify a scale of priorities per group and per performance, as the paper identies the evaluation needs of the stakeholder groups in supporting their decision-making processes in the training program. The main limitations of the study can be summarised as follows: . the analysis was based on a single training program, which reduces the possibility of generalisation; . the study focused on the pre-conditions for designing a stakeholder-based evaluation plan, not on the operational evaluation process; and . the analysis considered the attribution of importance by the stakeholders, without addressing the problem of how dimensions or latent variables can be translated into a set of essential and consistent indicators. In the light of these limitations, possible developments for further research could be: . The added value of stakeholder-based evaluation and the correlated levels of increase in training-process performances. This could be useful in identifying the applicability conditions for stakeholder-based evaluation and privileged contexts of application. . The impact of regulation systems (market and public policy) on the quality and quantity of stakeholder groups to be involved in the evaluation, their specic evaluation needs, and the evaluation process to be implemented. This potential development of the research should take into account the formalized evaluation systems based on quality and standards that are becoming more and more important, both in the market regulation system and the public-policy regulation system (i.e. European Credit Transfer and Accumulation System). . Methods for producing consensus among stakeholders and the process of creating a shared evaluation program. From a methodological point of view, research should integrate both qualitative methods (Talbot, 1992; Wills, 1993; Maxwell, 1996; Miles and Huberman, 1994) and survey research methods (Hinkin, 1998; Miller, 1994); and, in particular for the analysis of consensus building processes, it should be collaborative (Bramley and Kitson, 1994; Shani et al., 2007).

Notes 1. The European Union, for instance, has established a specic action programme (Decision No. 1720/2006/EC of the European Parliament) in the eld of life-long learning. The program included in the general policy Education and Training 2010 has the aim of contributing to the communitys development as an advanced knowledge society, in accordance with the Lisbon strategy objectives. 2. Table III illustrates the mean values of the responding sample, with the distribution in mean values $ 7.80; the mean values for all 35 items are presented in Appendix 1. 3. After identifying the existence of differences between the mean values, the post hoc interval test and comparisons of multiple couples make it possible to assess which mean differs from the others. The multiple interval tests enabled us to identify the homogeneous subclasses of means that did not differ from each other. By means of this multiple couple comparison, it was possible to identify the difference between each couple of means and obtain a matrix which ` s post highlighted the means of the groups with signicant differences (sig. 0.05), as in Scheffe hoc test used to analyze variances. It should be pointed out, however, that the test results used were affected by the low number of some of the stakeholder classes. 4. See Appendix 2 for a complete overview of the signicant differences between the mean values per stakeholder group. 5. In this case, in order to improve the interpretation of factors in the exploratory analysis, we decided to use a Varimax orthogonal rotation and therefore build independent factors. 6. The variables with factor loading , 0.5 were eliminated from the analysis (eight variables), as well the variables whose elimination improved the Cronbachs a value on the respective measure scales, making the same scale more coherent (one variable). 7. This learner care scale was built using three variables which loaded on the corresponding factor: although the Cronbachs alpha value was lower than the acceptable value (0.6), also for scales consisting of a reduced number of items, we decided to maintain the corresponding analysis for its theoretical signicance, as it corresponds to the rst level of evaluation of the ROI model.

Training evaluation

401

References Alliger, G.M. and Janak, E.A. (1989), Kirkpatricks levels of training criteria: thirty years later, Personnel Psychology, No. 42, pp. 331-42. Alliger, G.M., Tannenbaum, S.I., Bennett, W., Traver, H. and Shotland, A. (1997), A meta-analysis of the relations among training criteria, Personnel Psychology, No. 50, pp. 341-58. Bates, R.A. (2004), A Critical analysis of evaluation practice: the Kirkpatrick model and the principle of benecence, Evaluation and Program Planning, No. 27, pp. 341-7. Bramley, P. and Kitson, B. (1994), Evaluating training against business criteria, Journal of European Industrial Training, No. 1, pp. 10-14. Brandon, P.R. (1998), Stakeholder participation for the purpose of helping ensure evaluation validity: bridging the gap between collaborative and non-collaborative evaluations, American Journal of Evaluation, No. 19, pp. 325-37. Brandon, P.R. (1999), Involving program stakeholders in reviews of evaluators recommendations for program revisions, Evaluation and Program Planning, No. 22, pp. 363-72. Brown, D.C. (1994), How managers and training professionals attribute causality for results: implications for training evaluation, unpublished doctoral dissertation, College of Education, University of Illinois, Urbana-Champaign, IL.

JEIT 35,4

402

Cannon-Bowers, J.A., Salas, E., Tannenbaum, S.I. and Mathieu, J.E. (1995), Toward theoretically based principles of training effectiveness: a model and initial empirical investigation, Military Psychology, No. 7, pp. 141-64. Cousins, B. and Whitmore, E. (1998), Framing participatory evaluation, New Directions for Evaluation, No. 80, pp. 5-23. Donaldson, T. and Preston, L.E. (1995), The stakeholder theory of the corporation: concepts, evidence, and implications, Academy of Management Review, Vol. 20 No. 1, pp. 65-91. Flynn, D.J. (1992), Information Systems Requirements: Determination and Analysis, McGraw-Hill, London. Ford, J.K. and Kraiger, K. (1995), The application of cognitive constructs and principles to the instructional systems design model of training: implications for needs assessment, design, and transfer, International Review of Industrial and Organizational Psychology, Wiley, Chichester, No. 10, pp. 1-48. Friedman, A.L. and Miles, S. (2006), Stakeholder: Theory and Practice, Oxford University Press, Oxford. Garavan, T.N. (1995), HRD stakeholders: their philosophies, values, expectations and evaluation criteria, Journal of Evaluation Industrial Training, Vol. 19 No. 10, pp. 17-30. Gregory, A. (2000), Problematizing participation, Evaluation, Vol. 6 No. 2, pp. 179-99. Hinkin, T.K. (1998), A brief tutorial on the development of measures for use in survey questionnaires, Organizational Research Methods, Vol. 1, pp. 104-21. Holte-McKenzie, M., Forde, S. and Theobald, S. (2006), Development of a participatory monitoring and evaluation strategy, Evaluation and Program Planning, No. 29, pp. 365-76. Kirkpatrick, D.L. (1998), Evaluating Training Programs: The Four Levels, Berrett-Koehler, San Francisco, CA. Kontoghiorghes, C. (2001), Factors affecting training effectiveness in the context of the introduction of a new technology a US case study, International Journal of Training and Development, Vol. 5 No. 4, pp. 248-60. Lazarsfeld, P.F. and Rosenberg, M. (1955), The Language of Social Research, The Free Press, New York, NY. LeBaron Wallace, T. (2008), Integrating participatory elements into an effectiveness evaluation, Studies in Educational Evaluation, No. 34, pp. 201-7. Lewis, T. (1996), A model for thinking about the evaluation of training, Performance Improvement Quarterly, Vol. 9 No. 1, pp. 3-22. Mark, M.M., Henry, G.T. and Julnes, G. (2000), Review of Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs, Jossey Bass, San Francisco, CA. Mathie, A. and Greene, J.C. (1997), Stakeholder participation in evaluation: how important is diversity?, Evaluation and Program Planning, No. 20, pp. 279-85. Maxwell, J.A. (1996), Qualitative Research Design: An Interactive Approach, Sage, Thousand Oaks, CA. Meny, Y. and Thoenig, J.C. (1989), Politiques Publiques, PUF, Paris. Michalski, G.V. and Cousins, J.B. (2000), Differences in stakeholder perceptions about training evaluation: a concept mapping/pattern matching investigation, Evaluation and Program Planning, No. 23, pp. 211-30. Miles, M.B. and Huberman, A.M. (1994), Qualitative Data Analysis, 2nd ed., Sage, Thousand Oaks, CA.

Miller, T.I. (1994), Designing and conducting surveys, in Wholey, J.S., Hatry, H.P. and Newcomer, K.E. (Eds), Handbook of Practical Program Evaluation, Jossey Bass, San Francisco, CA, pp. 271-92. Nickols, F.W. (2005), Why a stakeholder approach to evaluation training, Advances in Developing Human Resources, Vol. 7 No. 1, pp. 121-34. Ostrom, E. (1990), Governing the Commons: The Evolution of Institutions for Collective Action, Cambridge University Press, Cambridge. Phillips, J.J. (1996), ROI: the search for best practices, Training and Development, No. 50, pp. 42-7. Polanyi, K. (1944), The Great Transformation, Holt, Rinehart & Winston Inc., New York, NY. Reineke, R. (1991), Stakeholder involvement in evaluation: suggestion for practice, American Journal of Evaluation, Vol. 12 No. 39, pp. 39-44. Ross, J.A. (2008), Cost-utility analysis in educational needs assessment, Evaluation and Program Planning, No. 31, pp. 356-67. Rossi, P., Freeman, H.E. and Lipsey, M.W. (1999), Evaluation: A Systematic Approach, 6th ed., Sage, Thousand Oaks, CA. Salas, E. and Cannon-Bowers, J.A. (2001), The science of training: a decade of progress, Annual Review of Psychology, No. 52, pp. 471-97. Scriven, M. (1993), Evaluation Thesaurus, Sage, London. Shani, A.B., Mohrman, S.A., Pasmore, W.A., Stymne, B. and Adler, N. (2007), Handbook of Collaborative Research, Sage, London. Talbot, C. (1992), Evaluation and validation: a mixed approach, Journal of European Industrial Training, Vol. 16 No. 5, pp. 26-32. Tannenbaum, S.I. and Yukl, G. (1992), Training and development in work organizations, Annual Review of Psychology, No. 43, pp. 399-441. Wills, S. (1993), Evaluation concerns: a systematic response, Journal of European Industrial Training, Vol. 17 No. 10, pp. 10-14. Further reading Abernathy, D.J. (1999), Thinking outside the evaluation box, Training & Development, Vol. 53 No. 2, pp. 19-23. Alkin, M.C., Hofstetter, C.H. and Ai, X. (1998), Stakeholder concepts in program evaluation, in Reynolds, A. and Walberg, H. (Eds), Advances in Educational Productivity, No. 7, JAI Press, Greenwich, CT, pp. 87-113. Bassi, L., Benson, G. and Cheney, S. (1996), The top ten trends, Training & Development, No. 50, pp. 29-33. Bates, R.A., Holton, E.F. III, Seyler, D.A. and Carvalho, M.A. (2000), The role of interpersonal factors in the application of computer-based training in an industrial setting, Human Resource Development International, Vol. 3, pp. 19-43. Bryk, A.S. (1983), Stakeholder-based Evaluation: New Directions for Program Evaluation, Jossey Bass, San Francisco, CA. Burke Johnson, R. (1991), Toward a theoretical model of evaluation utilisation, Evaluation and Program Planning, No. 21, pp. 93-110. Cook, T.D., Leviton, L.C. and Shadish, W.R. (1985), Program evaluation, in Lindzey, G. and Aronson, E. (Eds), Handbook of Social Psychology, 3rd ed., Random House, New York, NY, pp. 699-777.

Training evaluation

403

JEIT 35,4

404

Cronbach, L.J., Ambron, S.R., Dornbusch, S.M., Hess, R.D., Hornik, R.C., Phillips, D.C., Walker, D.F. and Weiner, S.S. (1982), Toward Reform of Program Evaluation, Jossey Bass, San Francisco, CA. Fitzenz, J. (1988), Proving the value of training, Personnel, March, pp. 17-23. Ford, J.K., Quinones, M., Sego, D. and Sorra, J. (1992), Factors affecting the opportunity to use trained skills on the job, Personnel Psychology, No. 45, pp. 511-27. Garaway, G.B. (1995), Partecipatory evaluation, Studies in Educational Evaluation, Vol. 21 No. 1, pp. 85-102. Geber, B. (1995), Does your training make a difference? Prove it!, Training, No. 3, pp. 27-34. Greene, J.C. (1988), Stakeholder participation and utilization in program evaluation, Evaluation Review, No. 12, pp. 91-116. Guba, E.G. and Lincoln, Y.S. (1981), Effective Evaluation. Improving the Usefulness of Evaluation Results Through Responsive and Naturalistic Approaches, Jossey Bass, London. Guba, E.G. and Lincoln, Y.S. (1989), Fourth Generation Evaluation, Sage, Newbury Park, CA. Holton, E.F. III (1996), The awed four level evaluation model, Human Resource Development Quarterly, Vol. 7 No. 1, pp. 5-21. House, E.R. and Howe, K.R. (1999), Values in Evaluation and Social Research, Sage, Thousand Oaks, CA. Kearsley, G. (1982), Costs, Benets, and Productivity in Training Systems, Addison-Wesley, Reading, MA. King, J.A. (2007), Making sense of participatory evaluation, New Directions for Evaluation, No. 114, pp. 83-105. McLean, G.N. (2005), Examining approaches to HR evaluation: the strengths and weaknesses of popular measurement methods, Strategic Human Resources, Vol. 4 No. 2, pp. 24-7. McLinden, D.J. (1995), Proof, evidence, and complexity: understanding the impact of training and development in business, Performance Improvement Quarterly, Vol. 8, pp. 3-18. McLinden, D.J. and Trochim, W.M.K. (1998), Getting to parallel: assessing the return on expectations of training, Performance Improvement, No. 37, pp. 21-6. Madaus, G.F., Scriven, M.S. and Stufebeam, D.L. (1986), Evaluation Models: Viewpoints on Educational and Human Services Evaluation, Kluwer-Nijhoff, Boston, MA. Mark, M.M. and Shotland, R.L. (1985), Stakeholder-based evaluation and value judgments, Evaluation Review, No. 9, pp. 605-26. Michalski, G.V. and Cousins, J.B. (2001), Multiple perspectives on training evaluation: probing stakeholder perceptions in a global network development rm, American Journal of Evaluation, Vol. 22 No. 1, pp. 37-53. Ostrom, E. (2000), Collective action and the evolution of social norms, Journal of Economic Perspectives, Vol. 14 No. 3, pp. 137-58. Ostrom, E., Gardner, R. and Walker, J. (1994), Rules, Games and Common-pool Resources, The University of Michigan Press, Ann Arbor, MI. Patton, M.Q. (1998), Utilization-focused Evaluation, 3rd ed., Sage, Beverly Hills, CA. Phillips, J.J. (1997), Return on Investment in Training and Performance Improvement Programs, Gulf Publishing, Houston, TX. Reason, P. and Bradbury, H. (2001), Handbook of Action Research: Participative Inquiry and Practice, Sage, London.

Scriven, M. (1996), Goal-free evaluation, Evaluation News and Comment Magazine, Vol. 5 No. 2, pp. 5-9. Shadish, W.R., Cook, T.D. and Leviton, L.C. (1991), Foundation of Program Evaluation, Sage, Beverly Hills, CA. Sridharan, S., Campbell, B. and Zinzow, H. (2006), Developing a stakeholder-driven anticipated timeline of impact for evaluation of social programs, American Journal of Evaluation, No. 27, pp. 148-62. Tesoro, F. (1998), Implementing a ROI measurement process at Dell computer, Performance Improvement Quarterly, No. 11, pp. 103-14. Vassen, J. (2006), Programme theory evaluation: multicriteria decision aid and stakeholder values, Evaluation, No. 12, pp. 397-417. Weiss, C.H. (1983), The stakeholder approach to evaluation: origins and promise, New Directions for Program Evaluation, No. 17, pp. 3-14. Ya Hui Lien, B., Yu Yuan Hung, R. and McLean, G.N. (2007), Training evaluation based on cases of Taiwanese benchmarked high-tech companies, International Journal of Training and Development, Vol. 11 No. 1, pp. 35-48.

Training evaluation

405

(The Appendices follow overleaf.)

About the authors Marco Guerci is a Researcher at the Department of Management, Economics and Industrial Engineering of the Politecnico di Milano. His research interests are focused on human resource management, especially on training and development evaluation. Marco Guerci is the corresponding author and can be contacted at: guerci@mip.polimi.it Marco Vinante is a Senior Researcher and works as a Consultant for public institutions and private organisations. His research interests are focused on education, labour policies and workfare systems; furthermore, his research activities are strictly related to the evaluation of public and social policies.

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com Or visit our web site for further details: www.emeraldinsight.com/reprints

406

JEIT 35,4

1. Alignment between training level of program and level of participants 2. Avoidance of content overlap in different courses 3. Increased investment in training by companies 4. Increase in public resources to be invested in continuous training nancing 5. Comprehensiveness of course catalogues offered by training providers 6. Knowledge/capacity that the trainer has acquired during the training program 7. Creation of a network among training providers, nancer and companies 8. Preparation of standard training catalogues by training providers 9. Preparation of standard training packages by trainers 10. Accessibility to training program (for workers/ employees) 11. Heterogeneous levels of course participants (from different work categories and/or professional proles) 12. Satisfaction with the courses organisation (venues, scheduling, classroom layout[. . .]) 13. Satisfaction with the didactics/training methods 14. Impact of the training program on company results 15. Increase in level of innovation within the company after the training program 16. Integration of course catalogues offered provided by training providers 17. Number of bureaucratic procedures imposed on participating companies 18. More knowledge acquired by training providers of companies training needs 8.38 7.95 8.57 8.52 7.71 7.19 8.38 7.33 7.05 8.29 7.38 7.81 8.95 8.33 8.10 7.81 8.29 8.48 7.50 8.00 8.00 10.00 10.00 9.00 9.00 8.50 9.00 8.00 7.50 7.22 6.89 7.67 7.22 8.11 9.11 8.56 8.11 7.78 8.89 9.22 8.50 9.22 7.50 7.11 10.00 7.89 9.33 6.33 7.67 6.33 6.33 6.33 6.00 6.67 8.67 8.33 8.33 7.00 9.67 9.00 8.00 9.11 9.67 7.95 6.36 7.86 7.95 7.00 7.09 7.82 5.95 7.91 9.09 7.95 7.86 7.18 6.09 8.23 9.00 9.50 10.00 8.67 7.56 9.00 8.33 7.00 9.67 8.45 7.86 8.68

Table AI. Attribution of importance to the dimensions

Appendix 1

Public Companies authority 7.53 7.39 8.42 8.08 7.21 6.95 7.66 7.26 7.13 7.71 7.08 7.47 8.53 6.79 6.92 6.92 6.61 7.92

Training providers

Enteprise association

Trainers Participants Mean 8.39 7.88 9.06 8.56 8.09 7.16 8.23 7.19 7.00 7.80 6.86 7.66 8.72 8.33 8.22 7.62 8.09 8.56 (continued )

Public Companies authority Trainers Participants Mean 8.59 6.86 6.55 6.41 6.23 8.50 7.14 7.00 9.36 5.55 6.09 9.00 8.67 9.11 9.00 10.00 9.00 7.67 8.22 9.67 9.33 7.67 7.33 7.67 7.45 8.50 6.82 7.91 8.77 8.23 7.05 6.68 6.24 6.63 7.00 7.50 7.00 7.63 6.00 7.29 6.89 7.26 6.68 7.74 7.84 7.26 6.61 7.74 7.32 7.24 7.30 7.05 7.98 8.12 8.19 8.63 6.55 7.72 8.20 8.61 8.24 8.43 8.35 8.14 7.29 7.81 7.48 6.67 7.19 7.90 8.33 8.33 8.71 5.43 7.19 7.90 7.57 8.48 8.29 8.48 8.48 9.00 10.00 10.00 9.50 8.44 7.50 8.50 8.00 10.00 9.67 8.89 7.00 9.11 9.67 8.67 7.33 6.67 8.50 9.00 9.00 7.78 7.00 9.00 7.50 7.44 7.33 7.50 7.67 9.33 7.50 7.89 7.33 8.00 7.56 6.67 7.50 9.11 7.33

Training providers

Enteprise association

19. Improvement in training providers image among companies 20. Improvement in the companys image with its workers/ employees 21. Improvement in the image of human resources with line management/workers/employees 22. Improvement in the enterprise association image with the companies/workers/employees 23. Improvement in the external image of companies participating in the training program 24. New collaborations between companies and training providers 25. Participation by small and medium-sized businesses 26. Possibility to dene training nancing procedures with the public authorities 27. Degree of knowledge and skills acquired by participants 28. Number of companies participating in the project 29. Compliance with project tender obligations 30. Synergies between the project and other nancing sources for training 31. Satisfaction level of companies with the training program 32. Transparency of mechanisms controlling access to nanced training services 33. Efcient use of resources by training providers 34. Utility of acquired knowledge and skills for the careers of the participants 35. Utility of knowledge and skills acquired by participants for their jobs on a short-term basis

Training evaluation

407

Table AI.

408

JEIT 35,4

Quality and amount of knowledge and skills acquired by participants Number of bureaucratic procedures imposed on the participating company Impact of training program on company results

Transparency of mechanisms controlling access to nanced training services

Improvement of training providers image with the company

Potential to dene training nancing procedures with public authorities

Table AII. Analysis of variance, post hoc test by Scheffe (Sig. , 0.05), means and differences between groups

Appendix 2
Trainers vs participants: 1.732 Training providers vs trainers: 2.798 Companies vs trainers: 2.195 Companies vs participants: 1.544 Training providers vs participants: 2.427 Companies vs participants: 1.792 Training providers vs participants: 2,506 Trainers vs participants: 1986 Training providers vs trainers: 2.667 Training providers vs participants: 2.667

Appendix 3

Rotated matrix 0.805691 0.758309 0.742195 0.735902 0.66691 0.558772 0.517354 0.770078 0.712282 0.691241 0.680329 0.663488 0.516438 0.798593 0.769497 (continued )

Improving competitiveness of companies and support to human resources Learner care

Components Promotion of fairness and image Training services Network stabilization offer

Impact of training program on company results Utility of acquired knowledge and skills for professional careers of participants Utility of knowledge and skills acquired by participants for their jobs on a short-term basis Increase in level of innovation within the company after the training program Degree of knowledge and skills acquired by participants New collaborations between companies and training providers Satisfaction level of companies with the training program Improvement in the external image of companies participating in the training program Accessibility to training program (for workers/ employees) Improvement in the training providers image with the companies Synergies between the project and the other nancing sources for training Participation by the small and medium-sized businesses Improvement in the company associations image with the companies/workers/employees Increase in public resources to be invested in continuous training nancing Creation of a network among training providers, nancer and companies

Training evaluation

Table AIII. Factor matrix

409

410

JEIT 35,4

Rotated matrix

Possibility to dene training nancing procedures with the public authorities Increased investment in training by companies More knowledge acquired by training providers of companies training needs Integration of the course catalogues provided by training providers Comprehensiveness of course catalogues provided by training providers Preparation of standard training catalogues by the training providers Number of bureaucratic procedures imposed on participating companies Efcient use of resources by training providers Alignment between the training level of the program and the level of the participants Satisfaction with the courses organisation (venues, scheduling, classroom layout[. . .]) Satisfaction with the didactics/training methods

Notes: Extraction method: analysis of the main components; rotation method: varimax with Kaiser normalization; explained variance: 65.5 per cent; the rotation converged with six iterations

Table AIII. Improving competitiveness of companies and support to human resources Components Promotion of fairness and image 0.631817 0.605377 0.601301 0.849739 0.830391 0.652659 0.60184 0.582032 0.743441 0.589429 0.549852 Training services Network stabilization offer Learner care