Evaluation

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)
p.1
11 EVALUATION
Dianne Allen 11 EVALUATION............................................................................................................1 11.1 Experience posing questions.................................................................................1 11.2 Evaluation Issues...................................................................................................5 11.3 Evaluation in the Literature...................................................................................5 11.3.1 A Current Starting Point Recognising a Way of Dealing with the Implications of Multiplicity of Form Guba and Lincoln........................................6 11.3.2 Deciding in My Case......................................................................................7 11.3.3 Appropriate Evaluative Criteria for Reflective Research of Practice.............9 11.3.4 Evaluation and Values: Values-in-Use.........................................................12 11.4 Facilitating Evaluation of Evaluation..................................................................13 11.5 In Summary: Converging on the Complex: Learning/Inquiry/Evaluation..........14 Appendix 11.1.1 Evaluation Thinking Experience Stories............................................18 Values in Acting, in Learning to Change...................................................................18 But What about the Values Bit? You Ask..................................................................18 Research and Evaluation Story 14 September 2001...................................................24 Bibliography...................................................................................................................28 From the processes of (1) asking myself what is the learning to change that I was looking for in the in-action testing of my professional development activity design, and (2) reflectively engaging with the literature of the field, I have discovered (adjusted my conceptual framework to include the understanding) that learning to change involves inquiry. In exploring the nature of the inquiry required to provide the valid information about what to change and how to change it, I have discovered that part of learning to change involves taking an action that is different, and that in deciding to act, a practitioner is involved in making a practical judgement, and that the process of making practical judgements, to be able to be improved, needs to be open to evaluation. In this chapter I am involved in exploring more closely what is the nature of evaluation.
11.1 Experience posing questions By comparison with learning to change, and inquiring into inquiry, my first response to presenting the experience that has posed the question/s about the nature of evaluation is to say that there is no evaluation story, and that is the story. The second thoughts response is that this investigation, as a whole, represents (some of) the evaluation story. There is a story walking hand-in-hand with the thesis task/ the
p.2
research activity, and it is, for me, the development of a growing understanding of what evaluation is. The experiences that may build up to one of the possible evaluation stories, in a brief, point-form, chronological list, are as follows: Evaluation in the library becoming dissatisfied with the process, the difficulty of measuring significant outcomes, 1978-1988 Becoming more involved in organisational policy and practice for personnel issues staff training policy, staff internal promotion/remuneration policy and practice, 1985-1996 Performance management and competency standards, and having the form but not the substance, 1992-1999; how the socio-technical analysis of skills (and competencies) was not likely to yield organisational effectiveness improvement, and that this process was a diversion of scarce resources, 1993-1996 Evaluation in the management plan, its use in management non-ownership; what is easy to measure becomes the focus; what is measured is what is then valued again, the distortion of what the organisation should really be about 1992-1999; this self-consciousness on the elements that are easiest to discern, 1997-1999 New design for the strategic plan raises the question of: are there better ways to evaluate?, 1997-1999 No obvious answers in the literature (Management and Human Resources literature, but only scanning material incidentally, no systematic investigation, not a clearly articulated, organisational problem to which I needed to attend until 1997; and nothing seen in passing during 1996-1998 while doing Master of Dispute Resolution assuming that the question, though embedded, was doing its strange attraction work); seminars and war stories about organisational change and development, and touted management fixes Management By Objectives, Zero Base Budgeting, Corporate Management for Local Government (1975/6, 1980-1985-1992) 19921999 Part of this study by research would need to include further examination of this issue, November 1998-February 1999
p.3
New input from the supervisors experience of the literature (R Mohr) and my reading of the same -> nothing new that I did not already know about, already have some misgivings about/ dissatisfaction with, April 1999
Experience Based Learning 1 at UTS, July 1999: self-assessment enunciation; own satisfaction criteria; easier to enunciate the goals and the nature of the evidence that would indicate those goals have been met for self-directed learning than to enunciate some assessment criteria
Reflective Research Of Practice, if it is an innovative model, may need an innovative evaluation, and so wanting to engage with that with an appropriate evaluation process, and starting to enunciate that as I understood it, August 1999
Solution 1, August-November 1999: Toulmin diagram; looking for triangulation: individual peers/supervisors possibly clients; self-awareness questions; before, during and after the intervention process; questionnaire design and conduct of Benchmark Questionnaire with ABE group in November 1999, and conduct of Progress Questionnaire with ABE group in February 2000
First round of report drafting January-February 2000: For me, the temptation to frame the questions and undertake this evaluation in typically traditional ways has been most seductive. Again, if there is a contribution to make by this research, it may well be how to design congruent evaluation which passes muster.
Group 1 (ABE) questionnaire application benchmark and progress application and comparison - ambiguous results, March 2000 Ambiguity in questionnaire results application 1 (ABE group) March 2000; plus sketchiness of data; plus no alternative group for comparison; plus clock ticking in thesis study frame -> interviews April 2000
Interview 1 round (ABE) analysis, June 2000 Availability of a second group (CNHS) June 2000 and capacity to compare ABE and CNHS, June 2000 December 2000 Input from UTS seminar on Contemporary Issues in Australian/Adult Education others are experiencing the difficulty of enunciating competencies and what is involved in the assessment of them and mapping these to an educational curriculum. This is what is required if work-based learning is to be used for, and recognised in, educational accreditation, July 2000
p.4
Research Proposal Review Presentation preparations What is evaluation? rears its ugly head again almost leads to complete transformation of Research Proposal, December 2000-January 2001
Processing of data from ABE and CNHS work, January 2001-August 2001 What I am valuing as having meaning for me new - Autonomy Lab workshop, June 2001; awareness of egocentricity in meaning making; live experience of a collaborative interaction amongst peers, facilitators, June 2001
Draft writing work August 2001 - values and learning - recognition that no attention has been paid to the literature on values in learning issue - documenting my thinking about this issue, August 2001 (See Appendix 11.1.1 for further details)
Writing draft, July and August 2001 too much on what basis might I cut it back? Evaluation of Reflective Research of Practice What might be the appropriate criteria? Supervision debrief, 4 September 2001 Research and Evaluation story draft, 14 September 2001 (See Appendix 11.1.1 for further details) Evaluation drafting September-October 2001 Supervision inputs (C Fox) Input from University of Wollongongs Vice Chancellors Symposium 4 October 2001 Universities do not necessarily know how to do it; have not articulated some of their criteria of scholarship and are being caught up into the same trap of form driving out substance, and of emphasis on the tangible as the less tangible, more difficult to enunciate and be transparent about, are discounted
Going out to the field University of Wollongong academic staff e-list 10 October 2001; ARLIST and ARMNET 7 November 2001 Feedback from effective interaction with Richard Badham 22 October 2001 first occasion where I think I have done my work justice Evaluation drafting and Dunn material inquiry feedback from supervisor R Mohr, 5 November 2001 Doing evaluation of student work for EDUT-422, 9-26 November 2001 Response back to field ARLIST and ARMNET, December 2001 CultureShift consultants end of year get together collaborative inquiry, critical reflection experience and valuing in a group of peers espoused; difficult to achieve in practice, December 2001
p.5
Engaging in associate work with EDUT-422, March 2002-June 2004 - student work assessment, interacting interpersonally with the students; assessment result challenge; assessing inquiry work through inquiry report writing
11.2 Evaluation Issues Again, as with learning, and inquiry, the review of these experiences indicates to me that I have an implicit sense of evaluation. That implicit sense of evaluation, when it has been made explicit, has mostly taken the form of the objective, the quantitative. However, there has been a growing dissatisfaction with the accepted quantitative processes, both for me, and, as will be seen, for others. As an implicit sense, until now, I have not been pressed to enunciate what it is, nor have I seen my way clear to construct something alternative to the traditional forms. It is now evident that this task seems to be the same as the task of learning to change: firstly, to become self-aware of what is my personal form of evaluation; secondly, to explore what the field suggests as ways of evaluating; and thirdly to examine my personal form of evaluation and the fields
suggested ways of evaluating in order to open the assumptions to critical scrutiny. Once this is done, I can then consider: do I need to change, or do I need to work on bringing change to the field, or both, and if so, how do I go about that, and what is my rationale for these proposals of principles and processes?
11.3 Evaluation in the Literature As I have found that learning is not monolithic, nor is inquiry monolithic, in a like manner evaluation is not monolithic. The nature of the phenomenon, its multiplicity of form, is hidden by the words with which we label it the singulars: learning, inquiry, evaluation. Here is part of the sociolinguistic distortion that I am learning to deal with in this learning journey. Further, while there is a discipline of evaluation, and literature associated, there is also natural evaluation, an embedded and tacit practice of all
p.6
people, and in its embeddness and tacitness, the assumptions underlying the evaluation, as well as good description of what it is, has not come to my attention as I have been reading. However, I also concede that until I know what I do not know, I am unlikely to see it if it is there.
11.3.1 A Current Starting Point Recognising a Way of Dealing with the Implications of Multiplicity of Form Guba and Lincoln In their Fourth Generation Evaluation, Guba and Lincoln track the history of evaluation as a discipline, and note the way in which the field has conceived of it, practised it, and gradually adjusted their understanding of it (Guba, 1989). For Guba and Lincoln the differences between kinds of evaluation and understanding about evaluation, particularly as practised in the educational field, are generational, and they identify four such generations: Measurement-oriented evaluation (p.22); Descriptionoriented program evaluation (p.28); Judgment-oriented evaluation where standards to be met are introduced (p.30); Responsive (constructivist) evaluation the new level, the fourth generation (p.38). For responsive evaluation the key dynamic is negotiation (p.8). Negotiation occurs as the participants to the evaluation engage with one another to: (4a) negotiate constructions (p.8); constructions which are (4b) shaped by values (p.8); and (4c) linked to context (p.8); (4d) where the shaping depends on the participants who are representatives of all stakeholders (p.9); (4e) which must have an action orientation (p.10); and (4f) where the evaluator interacts to respect stakeholders (p.10). Further, for responsive evaluation, Guba and Lincoln indicate that there are four phases which may be re-iterated and may overlap (p.42). This iterative process, together with the action orientation has the familiar ring of action research. Indeed, in the literature there is evidence of action research being used as a form of evaluation (Dick, 2000). A closer look at Guba and Lincolns four phases shows that they also acknowledge that, depending on the matters being evaluated, either quantitative or qualitative methods might be required. So the fourth generation of evaluation gathers in the other three generations, applying them as and when appropriate. Which is appropriate, and in what circumstances, is part of what the participants to the evaluation, the stakeholders, are
p.7
involved in explicitly considering and negotiating. What is resolved by negotiation (a form of peer dialectic if you will) is dealt with, and if action is the next step then the negotiated action is taken. What is not resolved by negotiation, either before any intensive investigation is conducted, or after the appropriate information is gathered, analysed and reported, becomes the focus of the second round of the evaluative process. It is the nature of the bone of contention that determines what information is collected. If there are competing claims, information is gathered to test those claims. If there is a range of concerns, and competing concerns, information is gathered to determine the extent to which such concerns are justified. If there is a range of issues, and competing issues, information is gathered to support or refute the various sides of the issues. That is to say, the methodology of fourth generation evaluation, responsive evaluation, acknowledges the need for, and provides for, the use of method appropriate to the investigation (Guba, 1989, pp.41-45). A similar position was expressed in the debate that was conducted by conference participants in 1990 about alternative paradigms. Heshusius noted a view that central to alternative inquiry is the position that one stays true to the phenomena one wishes to understand in their own context (Heshusius, 1990, p.198). Reporting on the conference debate, Heshusius noted that such a view was not unanimously held. Toulmin, in his consideration of the legitimacy of action research for the investigation of practice situations, voices the same sentiment, and points out that the history of this position is long, and its proponents well-regarded, though at times neglected (Toulmin, 1996). He remarks those who have worked within an Aristotelean tradition have insisted, rather, that different kinds of inquiry aim at particular, distinct kinds of knowledge, with methods of inquiry appropriate to the subject matter and interests at stake. (p.204)
11.3.2 Deciding in My Case One of the criteria informing my focus, in the case associated with the investigation of the effectiveness, or otherwise, of my professional development activity and reflective research of practice, is practical and practice-relevant. Patton nominates the following as criteria for practical evaluation: utility, feasibility, propriety and accuracy
p.8
(Patton, 1982, p.9). By utility he indicated useful to some audience; by feasible he indicated in political terms, in practicality terms, in cost effectiveness terms; by propriety he indicated conducted fairly and ethically; and by accuracy he nominated technical adequacy. Marshall identifies usable knowledge, transferable knowledge, validity, practicality, utility, credibility, worth (p.195-6) as elements of goodness criteria, while noting that they are also socially constructed judgment calls (Marshall, 1990). Whitehead and Schwandt question this searching for criteria (Whitehead, 1997-present; 2003; Schwandt, 1996). In Whiteheads case, I understand his challenge to be about who is judging whom about what, and what responsibility the practitioner has for making their own case about their own practice, and that certain processes associated with establishing criteria have more to do with hegemony and power-plays than with effective practice (Whitehead, 1997-present, 2003). Whiteheads focus on values, as being a significant component of professional practice, especially for teaching/ facilitating learning, is there in his threefold reflective question structure: 1. How do I improve my practice? 2. How do I help you improve your learning? 3. How do I live my values more fully in my practice?. In Schwandts case, I understand his challenge to be that the kind of thinking that associates with criteria, and indeed with orthodoxy in methodology, can be seen to be an inappropriate grasping for certainty in a world that is inherently uncertain. As such, the stance of striving for unitary orthodoxy represents ontological, as well as ecological, invalidity. It is an attempt to find a simple escape hatch in a complex world. And it will not do. It does not serve the practitioners need for further effectiveness (Schwandt, 1996). Another consideration then is what are the particulars in the case of my professional development activity design and of reflective research of practice? What I have been trying to work with is a model of inquiry that I am describing as reflective research of practice. Does it have some distinctive particularities that indicate that evaluation of a certain sort is needed, that evaluation of another sort is inappropriate? The first correction that I need to make is with the perception that one kind of evaluation will serve the purpose. My question needs to be phrased: Does reflective research of practice, as applied to my professional development activity design, have some
p.9
distinctive particularities that indicate that certain kinds of evaluation are needed, and that certain other kinds of evaluation are inappropriate?
11.3.3 Appropriate Evaluative Criteria for Reflective Research of Practice If, as noted in Chapter 4, and developed in Chapter 10.6, reflective research, based on Kressels concept (Kressel, 1997), is an innovative method of research for inquiry into practice, then it will need an innovative mix of methods, with their respective processes of evaluation. The nexus of the nature of the inquiry for the form of knowledge required, informs the choice of methods and the appropriate evaluation used in those methods. My first perception, of the possibility of there being a single congruent evaluation for a reflective research of practice study, has been adjusted. If there is a single overriding evaluation related to congruence, it is one of fitness, but I begin to sense my argument to be circular at this point. To answer what are the appropriate and congruent evaluations applying in reflective research of practice?, I have needed to go back to the literature of reflection and reflective practice and revisit the understanding of evaluation in the light of the difference being enunciated, and then I have needed to examine the assumptions involved in my and others resilient frames. That revisit, much of which constituted the rebuilding of my conceptual framework concerning the nature of inquiry, and has been presented in Chapter 10.3, involved considering (1) what was my logic, the basic structure of my argument for requiring congruent evaluations while conducting reflective research of practice; (2) what was my position within the spectrum of varieties of inquiry; (3) re-evaluating the argument and position of the key contributors to my conceptual framework; (4) revisiting conceptions of reflection and its role in learning and inquiry; (5) revisiting aspects of inquiry as they seek to manage bias and complexity; revisiting the literature on evaluation; and from those activities (6) preparing a summary table, of how these criteria apply in the case of reflective research of practice.
p.10
In considering what was my logic, what was the basic structure of my argument, for requiring congruent evaluations while conducting reflective research of practice, I identified that it was based on the principle of justice. The saying, let the punishment fit the crime, came to mind. Further, I identified that the element of logic, applied in order to perceive any absence of fit, may be a source of thinking/ argument error (Dunn, 1982), especially if I am dealing with concepts that belong to different logical levels. I then explored the extent to which authorities in the field address the concept of congruent evaluation, noting on the way, inputs from Mackay (1999); Toulmin (1996); Schon (1991); Mezirow (1991); Heron and Reason (1997, 2001); Heron (1996a, b, 1988, 1981a and b); Mellett (2000); Guba and Lincoln (1989); Guba (1990b); and Patton (2002). Much of this part of the argument has been enunciated in Chapter 10, in the discussion of the nexus of methodology-epistemology-ontology-axiology. Finally, I noted how critical theory informs the process: I recognised the justice of the immanent critique that there is an inherent contradiction if there is no match of fitness. My next step was to take stock of what was my position within the spectrum of varieties of inquiry. In doing that, I recognised the extent to which my attention is given to empirical material and the extent to which my attention is given to evidence from those practitioners who have had a continuous engagement, over a long period, with the issues of evaluation in-practice. Re-evaluating the argument and position of the key contributors to my conceptual framework required me to take into account Kressels description of reflective research (Kressel, 1997) and the discussions of methodological principles and issues provided both there and in the literature of the field where I was giving my attention: namely Argyris (1993), Mezirow (1991), Bateson (1979), Heron and Reason (2001, 1997); Heron, 1996a and b, 1988, 1985, 1981). I have discerned that I privilege the empirical, as do they. I have also discerned that for some of these authors, the experience of the dominant traditional view can still be part of what may be their resilient frame as well mine. It appears to me that such a resilient frame operates particularly in the case of Kressel and Argyris, and it takes the enunciation of the issues, as found in Herons work, to make some of the hurdles, in moving closer to more liberty and variety in methods, clearer.
p.11
Revisiting conceptions of reflection and its role in learning and inquiry has been necessary to identify what is distinctive about reflective research of practice, and I have enunciated how the field is conceptualising reflection and its role in learning and inquiry in Chapter 10.6. Such a revisit highlighted the beyond-cognitive/rational processes that need to be accommodated in the nature of reflective research of practice, and which consequently requires that appropriate criteria for evaluations in those components of reflective research of practice also move beyond the cognitive/rational realm. One way in which reflective work does that, is found in how it takes a practitioner into the realm of artistry and aesthetic judgement. But, generally, it would appear that this aspect of research practice is either denied, or not understood, and is found to be elusive by those who would seek to clarify and dimension it, partly because they may well be looking in the wrong direction. Revisiting aspects of inquiry, as it seeks to manage bias and complexity, helped me recognise the value of cooperative inquiry using reflective work (Heron, 1985, 1988, 1992, 1996, 1997, 2001); and what are some of the hazards of the self-study issue (Barnes, 1998; Pinnegar, 1998; Hamilton, 1998), especially reflexivity (Hammersley, 1995); and that for research of practice the processes of action research are most appropriate (Schon, 1983; Argyris, 1974; Schon, 1995; Lewin, 1946; The Action Researcher Reader, 1988; Sanford, 1970; Carr, 1986; Toulmin, 1996). Again, I have traversed some of this territory in detail in Chapter 10.3.8-9. In addition, one of the evaluative criteria that I seek to honour, in this process, comes from other considerations of what is practical, and from the usable knowledge area (Dunn, 1982, 1997; Ravetz, 1987; Argyris, 1993). Revisiting the literature on evaluation told me that there is a sense in which determining appropriate criteria for evaluation is an open question, and it is open to me to argue what is needful in this case (Guba, 1989; Heshusius, 1990; Toulmin, 1996; Patton, 1982; Marshall, 1990). Indeed, if I have understood him correctly, it is Whiteheads view that the determination of the relevant criteria for the professionals practice is part of a professionals responsibility. It is, as it has been, the difference between being professional and not being professional ie exercising an independent arbitration of ones own standards of practice (Whitehead, 1997-present, 2003). The relevant criteria can be shared amongst peers, and in an individuals practice the peer professional
p.12
standards represent and operate as a minimum standard. However, when the profession deals with the ill-structured, and unique, especially when such a professional is perhaps the expert in their field, at that point it is the professionals own criteria of what is good/ best practice, that operate. It is these criteria of good/ best practice that result in the leading edge expertise developed. I have reported my conclusions from revisiting each of the above elements in Chapter 10.6. In essence, there are multiple evaluations that need to be applied in undertaking reflective research of practice, and being alert to the need to match appropriate criteria with the relevant element of the phenomenon being investigated is part of competence in reflective research of practice.
11.3.4 Evaluation and Values: Values-in-Use My work thus far has brought me to a personal boundary of the non-routine, to yet more ill-structured questions, to awareness of what I do not know. Having reached that point, there is then the possibility for me to go into that realm. In the case of evaluation, and the nature of values, and of awareness of my own values and values-inuse, as distinct from espousable values which are incongruent with my actions, I still have far to go, in mapping the territory. In amongst the experiences that I cannot yet converge into a knot of relevance that becomes a useful experiential story, is the conceding, in Appendix 11.1.1, that although I had identified values-held as a possible barrier to learning to change (1993-6), when it came to working with the literature to write this report, throughout 1999-2001, I struggled with all the other literature about learning, including focusing on the context where I was seeking to be effective, and failed to see values, failed to look specifically for discussion on clashes of values, or values as barriers to learning. Furthermore, in Appendix 11.1.1, my experiential story of detail, of contemporaneous data capturing of the thinking processes undertaken in the course of writing, and writing for coherence, I have evidence of a significant recurrence of confusion as I have engaged in this study. Evaluation, as the etymology shows, is to do with values. According to Guba and Lincoln, the first expression of evaluation, in educational practice, as a discipline, was
p.13
focused on measuring, as in value represented in numbers. Its second expression was with description, where the limits of quantifying are conceded, and quality is described, and sometimes as a first step to dimensioning, the equivalent of quantifying. Its third expression recognised the process of judgement measuring either a quantity, or a quality, against some standard. Guba and Lincolns fourth generation begins to concede that the players, in evaluation, have values that inform their analysis and their decision making, informed by analysis, to undertake action. Further, that for each player in an activity and the evaluation of the activity, the values they choose to invoke to settle uncertainty, may differ between one another, and now for collaboration/ cooperation, the difference between these values needs to be negotiated, to some sort of consensus. The exercise of values, values-in-use, is operative in a method, in its analytical phase. It is operative in determining what kind of knowledge is being looked for. Also, the expression of values may well be part of a phenomenons nature it certainly is, when the phenomenon is humankind. It is in this way that I can see that Heron and Reasons call for axiology, the study of values, to be compounded with ontology, epistemology and methodology (Heron, 2001), has its merit, its reasonableness.
11.4 Facilitating Evaluation of Evaluation Beyond the case that I have made for what is required to facilitate learning to change, I have little at this point that I can add to the task of facilitating the improvement of the practice of evaluation. As my first response story indicates, I do not yet have a coherent knot of relevance that helps me in this task (Bateson, 1979, p.13). My second round of interactive reflective work, as I have engaged in the literature in Chapters 9-11, has helped develop my understanding of learning to change, of inquiring into inquiry and of evaluating evaluation, taking me to another edge of the non-routine, the illstructured, the indeterminate, uncertain. I begin to know more intimately what I do not know. From my experience, the best I have to offer involves recommending engaging in reflective research of practice. The first step is to become more self-aware, to be aware that a person is evaluating and to begin noticing what the person is doing when they evaluate. Capturing data about that will then allow the evaluator to look for patterns, of when what they do is successful, in their terms, and when what they do is not, as well as being attentive to what they are using as the evaluative criteria that
p.14
represents success, in their terms. With input from others they can also begin to compare what they do with the practices of others, to discern more or less effective processes. With input from others, the way they are thinking about evaluation, and what they do when they do it and why, can be questioned and challenged, asking them to reach into their unexamined assumptions. That is the start. Where and when it finishes is much less known, excepting that the experience of the past tells me that it probably will not be finished; it is not a finishable entity.
11.5 In Summary: Converging on the Complex: Learning/Inquiry/Evaluation In these last three chapters I have been exploring what is involved in learning, inquiry, and evaluation, especially as I understand them applying to the task of making the change involved in seeking to improve a professionals in-practice activities. As I have undertaken that investigation I have experienced times when the definitions and boundaries of the parts are clear, and times when they lose clarity and become interdependently permeable. When words in discursive text fail, when conceptual propositions cease to deliver simple assertions, there is a need to use another tool to work with concepts. At this stage of my career, the use of the Venn Diagram, representing the set of IS compared with the set of IS NOT, for an entity, can be used to explore and represent developing understanding. Using Venn Diagrams I can represent Learning, Inquiry and Evaluation as three separate circles: as the distinctness of the words convey three independent and different entities.
p.15
Learning Evaluation
Inquiry
Then by overlapping these circles I can represent aspects of one that are common to another, and aspects of one that are completely unrelated to another. I can visually represent that there is learning that is inquiry that is learning; that there is learning that is not inquiry and there is inquiry that is not learning, and similarly for learning and evaluation and inquiry and evaluation.
Then the positioning of the three relative to one another can generate eight distinct areas and entities: L-not-I-not-E; I-not-L-not-E; E-not-L-not-I; L=I-not-E; I=E-not-L; E=Lnot-I; L=I=E; and not-L-not-I-not-E
p.16
If, however the circles representing the entities are different in size, there are other ways of considering the relationships, and considering the sizes and the relationships asks questions about how I am understanding the nature of the complex Learning/Inquiry/Evaluation
p.17
To the extent that physiologically and neurologically there is no perception of feeling, sight, hearing, taste, smell, without difference, and that perception is the evaluation of difference, then one can say that there can be no learning, no inquiry without evaluation, that learning and inquiry are dependent on evaluation; that inquiring to learn is dependent on evaluation and that learning to inquire is dependent on evaluation, and yet somehow, sometime, we learn to evaluate, and what we learn we can change.
p.18
Appendix 11.1.1 Evaluation Thinking Experience Stories

As I have undertaken this study I have been forced to think about how I am understanding values and evaluation as they interact. I have received feedback from a supervisor that part of my process is to write to find out what I think. The following two items of drafting, including developments from note work to capture thinking, especially while walking, are indicative of my processes, and how they have contributed to the role of values in learning and being prepared to take action, and what, if any, might be the distinction between research and evaluation.
Values in Acting, in Learning to Change

By 16 August 2001 I had material drafted for the literature review on learning appropriate to my inquiries. The structure and content of my mobilisable understanding at that point was represented by the following headings: L1. Experience Issues; L2. Consideration Issues: L2.1 Nature of Learning and Relationship to Change; L2.2 Formative and Transformative; L.2.3 Meta Aspect; L2.4 Barriers; L3. Adult Learning; L4. Experience, reflective work in learning; L5. Professional Development; L6 Peer Group Learning; L7. Premises; L8. Learning and Kressel. As I was drafting and editing and drafting I realised that although I had identified the issue of the place that values might have in being a barrier to learning to change, I had not spent any time or effort in engaging with the literature on values in learning, or indeed writing about it in any quantity, let alone depth, in my drafting to that point. That posed the question: what role did I understand values/ valuing to be playing. As I worked with that I drafted the following:
But What about the Values Bit? You Ask

What indeed. It has almost disappeared from sight, from at least my sight. Its amongst the unexamined, the taken for granted. But not entirely. It was the clash with values, and the idea that the reflective research of practice helped me deal with practice in a way that acknowledged values, that has brought me to this research task, to this stage. It has been some/ (most?) of the motivating force behind this activity. And yet it keeps disappearing from sight. In my explication of the reflective research of practice model, as being a potentially significant way of addressing improvement of practice, I have used Venn diagrams to work up the ideas. Practice, what the practitioner is doing, is one set. (This is represented as a circle, designated red for danger, red for de Bonos emotional thinking, red a primary colour). Reflection, what the practitioner is thinking, is a second set. (This is represented as a circle, designated yellow for the second primary colour, yellow in de Bonos model is for positive speculative thinking). Research, systematic inquiry, how the practitioner goes about exploring practice and thinking, is the third set. (This is represented as a circle, designated blue. Blue in de Bonos model is for managing the process of going through the thinking modes of emotional (red), factual (white), positive speculation (yellow), negative risk assessing (black), and creative (green). It is the systematic structuring of thinking about a problem. Blue is also the third primary colour). The intersection of research on practice (blue with red -> purple) is the area that I designate as the development of expertise the systematic study of practice is likely to develop into expertise. The intersection of reflection with practice (red with yellow -> orange) gives the area that I designate as theory-in-use the more reflection done of practice the greater the clarity about the theoryin-use operating for the mediator. The intersection of research with reflection (blue with yellow -> green) is the area that I designate as creativity (following the de Bono system). This may be appropriate as systematic examination of thinking may mean trying out different metaphors or analogies
p.19
etc, to develop a refined model, or to happen on a creatively new model that more satisfactorily covers the facts. The intersection of all three primary colours red, yellow, blue (practice, reflection and research) and the three secondary colours, purple, orange and green, (expertise, theory-in-use and creativity) gives the tertiary colour brown muddy not clear icky. (The rule of thumb for the artist, from colour theory, is that mixing makes mud.) This composite tertiary, reflective research of practice, is where there is integrity, congruence I+C= Icky. When a practitioner knows what they are doing, and the thinking that informs that doing, by the systematic examination of their practice and thinking, then they are doing what they intend, and achieving what they expect, so they can be fully responsible for it integrity. Their actions have congruence with their values integrity and authenticity. Also, when the practitioner knows what they are doing, and the thinking that informs that doing, then they can enunciate it for another and so teach or transmit practical knowledge, transmit knowledge of practice. In working with action research, and models of investigating practices like education, I have been introduced to the threefold model associated with Aristotle, and Habermas. As I have sought to gather this model into my understanding. I have been working with the terms technical, practical and critical. And matching them with other terms: technical-content, practical-process, critical-context; or technical: know-what, practical: know-how, critical: know-why (purpose, theory-in-use, unexamined assumptions, values that inform). In dispute resolution studies I was introduced to a three layer model: with attitudes the outer layer, beliefs the middle layer, and values the inner core. Attitudes can be changed relatively easily. Advertising etc, shifts in social norms etc, allow for an attitudinal change. Beliefs can usually be articulated espoused theory. Beliefs can be changed, but they are harder to change than attitudes. Beliefs are open to persuasion to change. Belief change can be expressed in attitude change. Values, by comparison, are more difficult to change. They are often unexpressed, and taken for granted. They are often made explicit in actions when under stress, in a dispute. They are the bottom line beyond which an individual will not move. Hence, running into a clash of values in a dispute usually leads to an intractable dispute. So, my understanding of values, and values in the learning process, has tended to be muddled and muddy. But I have been exploring, I have been learning as I have been doing this research project. The hermeneutic spiral is operating in my thinking and understanding. Part of the process of the development of this understanding includes the process of writing. Part of the process of developing understanding is the reading of the written work of others. Part of the process of developing understanding comes with trying to engage with others in talking through what is understood. The following chronicles some of my experience, and my modus operandi, which has helped me move from a relatively muddled position to a clearer one. It is presented as is, with little or no reworking at this stage. That will mean it is a bit rough, but it may also be a bit more informative for being this rough. On 16 August 2001, I realised that although I had identified values as significant to the problem of change through learning, I have tended to take for granted that I know what I am talking about. I had identified this significance as early as 1995/6 with the work with the Negotiation simulations and role plays, and the written assignment and thinking about the management interaction training and its failure in some quarters. Although I enunciated that this was one of the areas of investigation in this research project, and committed that to writing in November 2000, I hadnt written about it in detail, I hadnt gathered together the literature information on it. By comparison, I had done this for the learning to change issue, for adult learning, for learning by and from experience and with reflective work, and what is specially involved in professional development and peer learning. And until 16 August 2001, I didnt have Values and Learning included in the outline of what I was drafting to. There is little or nothing about learning values, about learning about learning values in the work so far. I had become stuck on/with the writing. I had a general structure. I had realised that I had more than enough material for the thesis specification, and that I needed to select from that and write towards a clear and specific, bounded purpose. I had two questions which were being used to filter the consideration of what I wrote, and how. They were: 1. Why is this here?; and 2. What do I want this to say?. I had returned to the Learning bit, by way of doing some drafting of the Intervention Design bit, and some tentative work on premises, and looking again at what I thought I was to be writing about in the Research bit. What had appeared to be straightforward before, was now not so clear. It was muddled. I was
p.20
stuck. I was seeking to have some clear idea about what I should be focusing on. I had been re-reading the introductory remarks to this Learning section and doing some editing. I revisited some of the work of Dewey and Boud about the nature of reflection. I had noticed that the way I was phrasing my questions, which gathered up what it was that was stimulating this study, was in the terms that there were two issues: values and change. I had engaged in prayer, sharing that I was stuck, reaffirming that I was wanting to serve a useful purpose in this labour on the thesis. I had my afternoon nap. On waking, it was clearer that I needed to focus on the learning associated with values and with action to change. I took to my pencil and paper work. The following tries to capture the steps I took in enunciating my thinking and understanding. 1. The realisation is that what I am looking at is values, and learning, and action. So: values&learning, values&action, learning&action, and values&learning&action. (This is a similar process to the one I had taken with the exploration of Reflective Research of Practice breaking it down into its parts, to analyse and clarify what is there in each part; starting to then recombine them, and seeing what comes from the synthesis of concepts.) Compared with the material on learning and action, I have very little literature-supported information on values, on value formation, on the learning of values, on the role of values in learning. And where I am at, in the stage of work with this thesis, is such that it is going to pose some serious practical problems if I now have to start out on that journey. This is a familiar dilemma. It is one that has been faced often throughout this task. What has started in dispute resolution has moved into education. What is involved in the intervention includes material on person and personality, and cognition. Do I have to become proficient in psychology in the field of personality and the field of cognition? They are whole new fields, and very extensively developed. I read Mezirow. Do I now need to explore the field of linguistics to be able to do what I am wanting to do/trying to do in this process? If so, then reflective research of practice is not very practical. So, now, do I need to stop where I am at, and really get into the study of ethics, the philosophy of ethical reasoning to do what I want to do? So, using my practical screen, I now try to ask myself and answer for myself, some basic questions about what I do understand, what I do know, already, about values. What of this field, can I mobilise? I get ideas like: survival, like Maslows Hierarchy of needs. Survival leads to: the move to maximising pleasure and minimising pain So why arent we all hedonists? Then, the next thought is of time: there is a difference in pleasure and pain which is dependent on time. Short term pleasure may be risky, since the longer term outcome is pain and death, while short term pain may be acceptable since it leads to longer term pleasure/ survival. Then, the next thought is about relationships: one individuals pleasure may be at the expense of anothers pain, so in our socialisation, our social construction, which is designed to deliver on survival imperatives beyond the individual to the species, including caring for dependent offspring, forming a long term relationship so that you are protected when you are most vulnerable (childbearing and caring for the young dependent child) this leads to the values associated with collaboration (Yes I am a socialised woman, though never a physical parent) My focus is with intentional action. So how do values interact with intentional action? Intentional action is informed by values. But how informed? There are multiple values, and there is competition for priority, dependent on context. Intentional implies having control. In this case I enunciate control as the capacity to make a decision to give priority to one value of another, and to act on it. This leads me to enunciate what I mean by decision. And to note, that the process of decision, of determining which value has priority, now, in this context, is itself a valuing action. (Here is my meta- aspect again.) Then I try my thinking out on some real examples, practical, experimental: I do something intentional to please you (so you will do something to please me) -> the essence of transaction, of exchange: the expectation of reciprocity the moral obligation to have no debts; I do something intentional to not pain you (you will do something to not pain me); I do not do something, intentional, because it will cause pain to you (you will reciprocate by restrain in action). [17/8/2001 addenda RosemaryH bit in negotiation about being unconditionally constructive as the antidote to needing trust for a relationship] Then I start to look at the issue of being self-serving, and being other-serving, and the realisation that being other-serving can in fact be potentially self-serving if the other is inculcated on the value of reciprocity, and being no-ones debtor
2. 3.
4.
5. 6. 7.
8. 9. 10. 11. 12.
13.
p.21
14. Somewhere along in this process*, I have noted that values are inculcated both non-cognitively (first process) and cognitively (second process). (I usually mean non-verbally=non-cognitively; and verbally=cognitively). I have also been doing some work on what action is about, in relation to survival, and working from some first principles related to physical development from birth to adulthood. (*I was not watching what I was doing as I was doing it, completely. I can record this as my modus operandi, because I am closer to it than usual, and I have now developed my recording process for reflective work to allow for a bit more reliable reconstruction, but not a completely accurate reconstruction.) 15. I now start to string some words and associations together: pleasure principle -> epicureanism -> utilitarianism. Then I remember, that I do not necessarily know exactly what these words denote, in their technical usages. So, I turn to my Dictionary of Philosophy, and look at these, and incidentally follow up on some other words and concepts. I see what it has to say about values, about practical -> pragmatism, utility 16. It is time to take a walk. On the walk I start to think about change and sensation, or feeling. Sensation, feeling, is the basis of sense perception of pleasure and/or pain. Change is a part of the process, biochemically and neurologically light ->cell -> chemical change -> nerve impulse -> sight. And at the biochemical and neurological level it is the same process of change, whether it is pleasure or pain. It is interpretation that determines the difference. That interpretation is mediated by where the message is received, what neurones respond to that input, and where the message then gets taken, and which neurones these sending neurones have intimate connections with, that determines what action/s the body then takes (I have been recently reading Asimovs The Human Brain, and so this biological knowledge has been refreshed). 17. So: change is an intrinsic element of sensation perception 18. So: some change/ variety is an intrinsic aspect of pleasure, of pain. Which way does it go? How is it interpreted? The pleasure is that which contributes to survival, the pain is that which is a threat to survival. 19. Again, short term / long term issues are explored at the practical thought experiment level: ice cream is pleasurable, but all icecream ->lose the taste for icecream, because there is no variety to give change in sensation which is pleasure. Also, too much icecream (at the expense of other foods) -> blood sugar level high, obesity, malnutrition and death. So, the difference that generates pleasure -> encouragement to the physical system to get, if possible, a balanced diet -> health, life, survival 20. So those sensations, or feelings, that we name as pleasure are those which are likely to be life enhancing; those sensations, or feelings, that we designate as pain are likely to be life threatening 21. Therefore, Affect is the first sort, the primary sorting mechanism, in making a distinction, a discrimination, a choice between options, and it is made for survival purposes. The choice is a decision. The decision is part of motivation we move, we orient our activity, towards those things that give pleasure, that are likely to contribute to enhancing our life. 22. Now I move on to an elaboration of this. 23. BUT Apprehended Affect can be maladaptive, if it is faulty. What might be adaptive in one situation, say the dysfunctional family, might not be adaptive in another, say the collaborative workplace. Similarly, what is adaptive in a functional mature family might not equip one to operate effectively in a competitive working environment. 24. The Affect response formation is occurring in our earliest experiences, so it is not necessarily mediated by words, by symbols, so it is not necessarily cognitive. 25. AND Affect experience can be associated temporally (in time) with anything else that is around in the environment at the time of experiencing the Affect sensation. So, some things can be inadvertently associated with the pleasure or the pain affect, things that have no actual relationship with the cause of pleasure or pain. 26. This is the basis of conditioning where an outsider is using the connection deliberately to train/ condition a desired response. 27. This association building is part of our understanding of cause-and-effect. If there is an actual association between one event and an Affect experience, then there is a true cause-and-effect relationship. If there has only been a coincidental /contemporaneous association there is not a true cause-and-effect. But that doesnt stop a person thinking so, and this is the basis of superstition. 28. So, there is a difference between an actual threat and an anticipated (because associated in time on another occasion) threat. 29. The actual threat is real and needs to be responded to appropriately, otherwise there is the risk of death. 30. The anticipated threat may be real, and if real and responded to appropriately -> life.
p.22
31. But when the anticipated threat is not real, and the anticipation represents a false inference, then it needs to be tested to confirm that it isnt a real threat. 32. Finding out that one needs to test an anticipated threat to confirm its actuality, is part of the education of learning to confirm if a message is ambiguous. It is part of learning that some messages are ambiguous. It is part of the process of learning about how to learn from experience, and what to learn from experience, and the basis of scientific exploration. Part of science is the process of determining the actualness of a threat, and in that way dealing with superstition. 33. Now, Affect and Anticipation start to link in my mind with Herons model if anticipation is equivalent to his imaginal/ imagination concept/s. I have not been able to fully appreciate Herons model to date. It is a model that is qualitatively different from the Aristotle/Habermas model, because it has four distinct components. 34. So, Anticipation/ the imaginal is the second sort, the secondary sorting mechanism, in making a distinction, a discrimination, a choice between options, and it is made for survival purposes. It uses time and mind directed to a survival purpose. Envisioning, having implications understanding (which may be a faulty cause-effect relationship), having foresight (if accurate = match with reality) has a survival value. 35. But if Anticipation is faulty it also acts to prevent an action, and so limits options that might be of survival value. 36. So, the next level in our hierarchy (according to Heron) is the development of the Conceptual. Part of the Conceptual is the understanding we have about cause-and-effect, as a concept, an abstraction, having rules of regularity, of relationship that apply, which have been tested over time, and found to be generally reliable, and so to be of survival value. 37. So, with the Conceptual, we start working on the inferences. That means we are able to do the testing of cause-and-effect relationships in our mind, rather than in actuality, and to get a reasonably correct answer. This is a third mechanism, a third level, a third order of being able to make a distinction, a discrimination, a choice between options, that is made for survival purposes. It is based on the potential of an action, a response to a stimulus, if carried through, to be of survival value. 38. SO: I get the following table of yes/no, stop/go considerations involved in taking intentional action Category: Affective: Imaginal: Conceptual: Practical: + + + GO ACTION Status: + = yes, survival value; 0=maybe; - = No, survival threat +/0 +/0 +/0 Note 2 +/0 +/0 - Note 1 + NO ACTION NO ACTION NO MAYBE ACTION ACTION 0 0 0 UNLIKELY TO BE ACTION
Note 1: this last process, where the conceptual is explored and determined at -, represents a relatively rare procedure for most people; it is intentional, and conscious, and involves something like the 4 phase analysis. It is the resort of a thoughtful person compared with an impulsive person. And it requires training to ensure that the thoughtfulness is effective, because there is such a thing as faulty thinking. Note 2: it doesnt matter whether these categories register as yes, no, or maybe, the prior sorts in the hierarchy rule. 39. Now, I explore another issue: change and regularity, and thinking about cause-and-effect. The basis of change and regularity: there is day and night change and regularity; there is the lunar month change and regularity; there is the year with its seasons, change and regularity. These cycles inform our understanding of time, and as such contribute to our concept of anticipation. The regularity of some things in our actual world and our experience of the actual world informs our ideas of the principle of uniformity, our tendency to generalise. Not everything is all change, totally unique. If it were all change, we would die from overexcitement, the overload, the exhaustion of forever needing to respond. This is part of the stress of change when it is happening more quickly than our usual expectation of reasonable timeframes. 40. The regularity is the security base from which we can venture out and explore our universe. The approximation (and it is an approximation) that not every thing is totally unique, is a fundamental of our system of understanding that there is sense in our life/ world, and how we make sense of this world.
p.23
41. Now, I take another approach. Can I apply another frame of understanding, which is not dependent, as the past has been, on adaptation for survival natural selection concepts? Here, I privilege the biblical view. How does the Bible deal with values? What does it have to say about SURVIVAL as A, as THE value? 42. What comes to mind is: the commands in Genesis be fruitful and multiple and subdue the earth a positive command to live. Then: but of the fruit of the tree of the knowledge of good and evil you shall not eat, in the day that you eat thereof you shall surely die a positive command to take proper cognisance of a threat to life, namely, death. And: What shall it profit a man to gain the world, if he loses his life, his self, his eternal soul. And the comment in the Psalms about if you knew when you were going to die, it would change your view -> wisdom -> the selection of right priorities about relative importance in day-to-day choices. 43. The next thrust is to explore: How are values related to the theory-in-use concept? Are we talking about the same thing? Both tend to be hard to enunciate. Both have clearest expression in action, especially in a stressful and/or dispute situation. I look at what I understand to be Argyris major theory-in-use that generates Model I behaviour, namely the issue of trust. I cannot trust another to look after my interests, so I take what action I think is needful to protect my interests I compete (eg for resources). I cannot trust another to look after their interests, so I take unilateral action to protect them, by the tactful action of withholding valid information which might threaten or embarrass them. The argument behind this is: the only one capable of looking after my interests and the interests of others is me. The looking after interests is a value, and it is the value of giving the correct priority to the things I value most, my values, my self-identity. This is what is at the basis of the Model I behaviour. 44. Another theory-in-use from Kressel it takes two to tango something that suits the concept (identity) of the mediator being a third party neutral, not being biased in their interventions. On first sight, the theory-in-use about others behaviour is not based on anything in my values. On second sight it is, if consideration is given to the mediators self-concept related to bias in practice. So, perhaps yes, perhaps, no, or maybe this is still open. I really do not know much about theories-inuse; and certainly not enough to identify one which does not involve some aspect of self-interest, self-identity. And my inclination, my impression from experience so far, about the ideology critique mindset of exploring intentions, and hidden social and political assumptions, is that I will probably find something there on every occasion. So: do I need to be skeptical and cynical about others behaviour? And how useful is that as a survival value? And survival and life in the short term and in the long term? 45. This is the point where this round of thinking, etc, comes to a close. 46. Then I note, as I re-read the Introduction to Howard Gardners Frames of Mind 2nd edition, that on p.xxii-xxiii, he says Constraints research has revealed that, by the end of early childhood, youngsters have developed powerful, and already entrenched theories about their immediate world: the world of physical objects and forces; the world of living entities; the world of human beings, including their minds. Surprisingly, and in contradiction to the claims of the great developmentalist Jean Piaget (Mussen and Kessen 1983), these nave conceptions and theories prove difficult to alter, despite years of schooling. And it so happens that the mind of the five-year-old ends up unaffected by the experiences of school. It is hard enough to teach even when anything can be taught; what to do if there are distinct limits and strong constraints on human cognition and learning? 47. So, now I am in the position of being alert to the question of values in learning. Now, I expect that as I read other authors, and re-read some of the authors already read, that I am likely to see values talked about, in ways that I didnt notice before, since I wasnt selectively attentive to it in the past. 48. Now, I come back to the question of action learning. If my model of Affect is + AND Imaginal (Anticipation) is + AND Conceptual is + is what are the necessary conditions for action, then Herons approach in education, of the reverse cycle of (1) Affect; (2) Conceptual; (3) Imaginal; (4) Practical may be well worth exploring again. 49. If this is the case for experiential learning, then Affect for the adult is the first port of call. It is presenting that this is of value for you, for survival. Survival for the adult, in the workplace, is now increased competence in meeting adult expectations; having increased skills in dealing with presenting problems; in being able to discharge adult responsibilities to ones own, and the satisfaction of others with whom the adult is interacting. The Affect, being able to feel pleasure (or take away the pain) is what is dictating the demands for relevance, for timeliness, for material, content or processes that is useful and useable in dealing with the presenting problem; it is this which is the significant part of the adults motivation to learn; it is this that determines the adults capacity to change, to work at learning to change.
p.24
50. What looks good on paper needs to be revisited through the screens of Affect, Imaginal and Conceptual and analysed: it is +; or 0; or -? It is the idiosyncratic pre-5-year-old conditioning that is informing the Imaginal anticipation of hurt which is the barrier to learning to change. So, now, the work is to use the Conceptual persuasion and other work to move the Imaginal from to 0 to + . Then the person can be taken to the experience and come out with the Affect moved from to 0 and then to +, because they have experienced that this action does not actually hurt. And if the action is an effective choice, then there may be more positive outcomes which means the actor has the experience of pleasure from the action. Now, I might be more able to return to the task of writing the other parts of this section, writing the other parts of this thesis. Using this as part of what I am to be focusing on. What has happened is that I have invoked the method/ process and privilege of action research. Of being able to go into a situation, say an intervention, where I am seeking to develop more understanding, with a fuzzy idea. I plan an action on the best advice available at hand at the time of going into the intervention. I undertake the action. There is a response. I now turn my attention to thinking some more about that response. This thinking results in a slightly less fuzzy idea which I take into the next round of work of trying to understand what is going on, in the intervention, in trying to understand what are the necessary and sufficient conditions needed to assist and adult to take an action to bring about change (for improvement) in their practice.
Then, as drafting proceeded over the following month, and I was party to discussions with post graduates about methodology, about theoretical frameworks, came the occasion of some more writing to express my thinking.
Research and Evaluation Story 14 September 2001

Research and Evaluation Story || Evaluation and Research (0830 to 0930 14/9/2001) Yesterday I ended up confused/ muddled again. The question was to consider what I was doing, or trying to do, in the chapter on evaluation. I had my blue chits from previous work on the other parts, and some earlier blue chits for evaluation. Now with a blank form a blank. After a sleep after lunch, I set to approach the matter in another way. To draft into a table all the original 23/7 etc blue chits, and see what that told me, ie to get that material back into my mind, and fresh. Reading it was not doing the trick, and I had had a reasonable break from the flurry of drafting to that structure that occurred between 23/7 and 20/8. So I set up a table with 5 columns. I typed in the Kressel bit. I copied and pasted the Learning and Research bits from previous work. I typed in the Intervention bit. I now collected the material from the recent summary work (date 6&7/9). The copy and paste, and minor typing individualisation was done. Then I had a blank box for the Evaluation bit, the why do I want it here; and the what do I want to say. I thought about it from scratch. I came up with: Investigation, to learn to change requires a decision to act. The basis of that decision to act needs to be as sound as possible. How we evaluate soundness to inform such a decision is then part of the process What say: the process of evaluation can be considered as a contested field This research is a different meaning scheme This research should be evaluated by appropriate criteria This is how this investigation should be evaluated This is the reason supporting this claim
p.25
There wasnt any purpose, as I could see it, in proceeding into the subsections. So I stopped and printed out the resulting document. I then noticed: that the dumping and minor editing of the summary bit indicated that what was now here was not what was in my mind. In my mind was the structure of having the literature up front, as per Kressel. Kressel first, DLA next, literature and Kressel interaction next. So in Learning and Research: relevant literature first; DLA stories next; DLA conceptualising next; premises, etc next. The dump and minor editing process indicated that the sentence in the summary that brought those three parts together and spoke descriptively of the Kressel bit was in fact conveying an inaccurate indication of what would be happening in the other two bits. I also noticed that what was there for the Evaluation bit for the why there and what say had almost nothing to do with the summary statement of 6&7/9: Then consideration is given to the issues of the appropriate criteria and method/s for evaluating (1) the effectiveness of the intervention, and (2) the practicality of reflective research of practice as a method of practice improvement. Oh, the power of the blank box!; the power of the structuring questions! The power of getting it on the same page to see it all together! Why were they so different? I was asking why evaluation is a significant part of this study. I was answering that from a different point of view from how I was seeing this chapter as being needful as an enunciation of what was going on in this thesis and how. I was answering that question from the context of the wider thesis as a whole. The view of the chapter from the summary was simply an arguing of the case for how to evaluate this thesis, and evaluation in this case. I now saw that evaluation is part of the decision to act or not. If evaluation is positive -> action. If the evaluation is negative -> no action (back to my values enunciation?? 14/9 thinking as drafting this story!). To make a change includes a decision to act. The basis of a decision to act is an evaluation. I was also continuing to be reacting to something my supervisor had said last week about judging and my being quite judgemental but that not quite coming through in my writing style; and a bit of the discussion of Monday night that developed around that theme, and in interaction with another students work on research about developing policy for enactment (PNG education issue). I noted: Am I, are others, getting into a muddle? Is evaluation research, is research evaluation? Evaluation involves some inquiry external, objective, within an accepted rationale eg scientific, to apparently manage what is otherwise usually unmanageable differences in values and how to resolve differences in values which is not by a distorted power play. (14/9/ as drafting the power play is quicker, so apparently more efficient than the time required to negotiate the Woolies and fixed prices versus the market and bartering of the third world issue again). The emperor (of scientific) has been found to have no clothes, or not any different clothes from others, eg power play. [Addendum 15/9: Do I need to explain this? The emperor = scientific method, traditional dominant empirical model, etc. The simplification that investigation can be objective, its conclusions are valuefree. It is not value free. One of the values is to minimise bias in certain areas to increase reliability. And that value, when this kind of evaluation is used in investigation in the workplace/ management context, as it is now the dominant given priority over other values. And it does this by the exercise of the power of the hegemony rather than proving its worth by gaining contemporaneous dialectic consensus amongst the stakeholders. We have accepted the model and its priority of values, on the basis of its credentials in other areas and on the mistaken argument of getting on with business; of the acceptance of the fixed price as being the fair price, and non-negotiable, so we can move onto something more important. Because we have devalued the process of negotiation, of crafting the contemporaneous dialectically formed consensus. (to use the terminology of the literature). 18/9: It is in fact the value of doing things over against being and quality of relationship. It is the acceptance of a previous authority and consensus (tacitly formed probably, and to what extent did something like WW1 & WW2 have a role to play here?) In the present anti-authoritarianism, it is starting to be challenged again.]
p.26
So, Guba & Lincolns responsive evaluation is (only? 14/9) an endeavour to return to the more ethical basis of evaluation the negotiation of differences in values and the construction of a consensus that we can agree to live with I note also that Toulmin looks at reasoning in a variety of fields: Legal, science, the arts (aesthetics), management, ethics. His point is that there are different ways of reasoning which is appropriate to different fields! The Aristotle versus Plato (high-Geometry) debate again! So I look again: what is research? Research/Inquiry? It is: To find new knowledge To test claims/ hypothesis for reliability = we can trust this to act on it (AND HERE IS THE LINKAGE TO EVALUATION) Eg calculations in engineering to hold up a bridge; genetic understanding -> treatments for systemic disease. So, investigation is: collect data; build a model to inform action! And the linkage between the why it is here, as drafted as: Investigation, to learn to change requires a decision to act. The basis of that decision to act needs to be as sound as possible. How we evaluate soundness to inform such a decision is then part of the process And the What say, as drafted as: The process of evaluation can be considered as a contested field This research is a different meaning scheme This research should be evaluated by appropriate criteria This is how this investigation should be evaluated This is the reason supporting this claim This linkage is not logical; there is no relationship between the one and the other. The what say was reflecting the material from the xx/ summary. So I had more thinking to do. What am I wanting to say in the Research bit and why? What am I wanting to say in the evaluation bit and why? Where does the material go? separate chapter, or: some in introduction, some in research; some in intervention? Im back to square one. What do I mean and understand by research? What do I mean and understand by evaluation? If my thinking can be faulty and distorted, so can others. If others dont pick up my errors, then they may not pick up the errors of others. Then all we are/may be doing, in writing academic stuff, may be perpetuating errors; compounding errors and disseminating as gospel the errors of others. [Addendum 15/9: We do this by taking and using them as the authority, the grounds and warrants of our arguments.] I also note: Knowledge usable aspect for me. [Addendum 15/9: I overlooked this in drafting first time. Now that I see that and now that I am looking with an eye to underlying story, primed by re-reading Schons Reflective Turn, p.343-359, I see that one of my underlying stories is the value that knowledge is for use. I think that knowledge for knowledge' sake, if it is valid and reliable, becomes, in time, useful. 18/9: I also recognise that the valuing of cultural perspectives may mean that sometimes it gets lost for a time. I also note, that in drafting the chapter material and getting caught up with Reflection and Reflective Practice, has meant that the practical focus has disappeared from view and only returned a day after, having left it aside.] Reflections: 14/9/2001 Of 13/9
p.27
Thinking about draft design: Why here; What say hold up at first; then try approaching it from another angle a table and typing in the others and compare -> question: is evaluation research?; and the reverse: is research evaluation?; and if evaluation is research and research evaluation, or especially the first one, then evaluation of research is research of research a meta- aspect again; does this explain why I keep on getting confused? it is the logical levels error of Bateson/Russell again.
0800 while out on walk do I write up my story of evaluation & research as an illustrative story for the thesis. If I do, how detailed do I go? If I do, the how and why this gets picked is the confluence of writing and the reading and the point in the thesis drafting, and the level of awareness of the issue at this stage. Ie it is incidental/ a contemporaneous circumstance; and catching it while it is new and fresh, and hasnt subsided into memory and/or been reprocessed. Even so, it represents some reprocessing overnight while sleeping; considering while on the walk; what I am now doing as I try to construct the memory as distinct from a stream of consciousness attempt and anyway what is a stream of consciousness attempt a la V Woolf? And now I am self-conscious about this process, and self-conscious in the way of am I only creating data to prove my point; and of the supervisor's challenge to stop collecting data and start processing it. But the writing element is part of the process of research .. so I will collect and write this story now to get the contemporaneous record, whether or not I use it in the thesis. 13/9/2001 1100 no progress on thinking about draft design 12/9/2001 3.30am: Woke reasonably refreshed and with some clarity about evaluation bit: what I have tried and when; what has happened; now I can perhaps answer the question about whether I have learned; there was intention to learn; there was intention to implement appropriate; but is also faded into the background in intervention action practice; then the intuitive evaluation took its role Thinking some more about quantitative/qualitative split that is part of methodology which comes first ontology, epistemology, methodology in any structural analysis/ arrangement challenged by one of the PGs sharing that the visual helps To what extent the visual figures help people with the text I have just found my intervention map of 12/99 what was the thinking then This raises the issue of thinking and learning Mezirows distinction thinking has happened, learning has not happened This raises the issue of material from tacit -> thinking; then returns to tacit -> intuitive actioning? to what extent does the data show this? To what extent will I see this now I am looking for it!? Back to thinking about method/onto/epistem/ologies: Qualitative = describe = enunciate; for human research on the actions of humans (individual=psychology/autobiography; group=sociology/politics/history) the thinking-meaningunderstanding- reasonableness informing intentionality (therefore ethical questions can be asked) we need to note what we are describing/ enunciating is their construct. The extent to which I can enter their construct, to claim understanding accurate interpretation, to understand their experience, to learn from it (vicariously) is constrained, and needs to be worked at. The construct is framed by language and the metaphorical base of their (and my) culture. The need to explore assumptions but enunciating assumptions is not easy; not usually done; when enunciated might be espoused rather than in-use .. Structural analysis chicken and egg Ontology=nature of being Epistemology=nature of knowing (knowing is an outcome of nature of being) Methodology= how? How get to know (determined by what understand knowing to be; determined by nature of beings capacity to know) so hierarchy: O/E/M Looking at history of living world classification, we work from the distinctly different individual animals (methodology) by asking questions about similarities and differences -> most general differences (animal/vegetable) ||| Ontology -> and then we look at the inbetween similarities and differences phylum/genus (epistemology) ??
p.28
So the process would appear to be M/O/E ?? Is it worth working some more on this to make a picture? check original research 1998 thinking for overhead quantitative/qualitative split (old chemistry)
Bibliography
The Action Research Reader ( 3rd ed.)(1988). (Melb): Deakin Univ Pr. Argyris, C. (1993). Knowledge for Action: A Guide to Overcoming Barriers to Organizational Change. San Francisco: Jossey Bass. Argyris, C., & Schn, D. A. (1974). Theory in practice. San Francisco: Jossey Bass. Barnes, D. (1998). Foreword: Looking Forward: The Concluding Remarks at the Castle Conference. In M. L. Hamilton & S. Pinnegar & T. Russell & J. J. Loughran & V. K. LaBoskey (Eds.), Reconceptualizing Teaching Practice: Self-study in Teacher Education (pp. ix-xiv). London: Falmer Press. Bateson, G. (1979). Mind and Nature. New York: EP Dutton. Carr, W., & Kemmis, S. (1986). Becoming Critical: Education, Knowledge and Action Research (rev ed ed.). (Melb.): Deakin Univ Pr. Dick, B. (2000a). AREOL - Action Research and Evaluation On Line. URL http://www.scu.edu.au/schools/gcm/ar/areol. Available: http://www.scu.edu.au/schools/gcm/ar/areol. Dick, B. (2000b). The change process and action research. Session 2 of Areol- action research and evaluation on line. Available: http://www.scu.edu.au/schools/gcm/ar/areol/areol-session02.html. Dunn, W. N. (1982). Reforms as Arguments. Knowledge: Creation, Diffusion, Utilization, 3(3), 293-326. Dunn, W. N. (1997). Probing the boundaries of ignorance in policy analysis. (Initiating Change: Theory and Practice). American Behavioral Scientist, 40(3), 277-299. Guba, E. G., & Lincoln, Y. (1989). Fourth Generation Evaluation. Newbury Park, CA: Sage. Guba, E. G. (1990b). The Alternative Paradigm Dialog. In E. G. Guba (Ed.), The Paradigm Dialog (pp. 17-27). Newbury Park, Calif.: Sage. Hamilton, M. L., Pinnegar, S., Russell, T., Loughran, J. J., & LaBoskey, V. K. (Eds.). (1998). Reconceptualizing Teaching Practice: Self-study in Teacher Education. London: Falmer Press. Hammersley, M., & Atkinson, P. (1995). Ethnography: Principles in Practice ( 2nd ed.). London: Routledge. Heron, J. (1981). Philosophical basis for a new paradigm. In P. Reason & J. Rowan (Eds.), Human Inquiry: A Sourcebook of New Paradigm Research (pp. 19-35). Chichester: John Wiley & Sons. Heron, J. (1981). Experiential research methodology. In P. Reason & J. Rowan (Eds.), Human Inquiry: A Sourcebook of New Paradigm Research (pp. 153-166). Chichester: John Wiley & Sons. Heron, J. (1985). The Role of Reflection in a Cooperative Inquiry. In D. Boud & R. Keogh & D. Walker (Eds.), Reflection: Turning Experience into Learning (pp. 128-138). London: Kogan Page. Heron, J. (1988). Validity in Co-operative Inquiry. In P. Reason (Ed.), Human Inquiry in Action: Developments in New Paradigm Research (pp. 40-59). London: Sage. Heron, J. (1992). Feeling and Personhood: Psychology in Another Key. London: Sage. Heron, J. (1996a). Co-operative Inquiry: Research into the Human Condition. Thousand Oaks, Calif: Sage. Heron, J. (1996b). Helping whole people learn. In D. Boud & N. Miller (Eds.), Working with experience: animating learning (pp. 75-91). London: Routledge. Heron, J. (1996c). Quality as Primacy of the Practical. Qualitative Inquiry, 2(1), 41-. Heron, J. (1999). The Complete Facilitator's Handbook. London: Kogan Page. Heron, J., & Reason, P. (1997). A participatory inquiry paradigm. Qualitative Inquiry, 3(3), 274(221). Heron, J., & Reason, P. (2001). The Practice of Co-operative Inquiry: Research 'with' rather than 'on' People. In P. Reason & H. Bradbury (Eds.), Handbook of Action Research: Participative Inquiry and Practice (pp. 179-188). London: Sage. Heshusius, L. (1990). Discussion on Criteria. In E. G. Guba (Ed.), The Paradigm Dialog (pp. 198-201). Newbury Park, Calif.: Sage. Kressel, K. (1997). Practice-Relevant Research in Mediation: Toward a Reflective Research Paradigm. Negotiation Journal, 13(2), 143-160. Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34-36.
p.29
Mackay, H. (1999). Words that are lovely, dark, and deep. Sydney Morning Herald, Spectrum(4 September 1999), 22s. Marshall, C. (1990). Goodness Criteria: Are They Objective or Judgement Calls? In E. G. Guba (Ed.), The Paradigm Dialog (pp. 188-197). Newbury Park, Calif.: Sage. Mellett, P. (2000). Educational Action Research within Teaching as a Research-Based Profession. Mezirow, J. (1991). Transformative Dimensions of Adult Learning. San Francisco: Jossey-Bass. Patton, M. Q. (1982). Practical Evaluation. Calif.: Sage. Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd; rev ed of 2nd ed Qualitative Evaluation & Research Methods (1990); 1st ed 'Qualitative Evaluation Methods' 1980 ed.). Thousand Oaks, Calif.: Sage. Pinnegar, S. (1998). Part II: Methodological Perspectives: Introduction. In M. L. Hamilton & S. Pinnegar & T. Russell & J. J. Loughran & V. K. LaBoskey (Eds.), Reconceptualizing Teaching Practice: Self-study in Teacher Education (pp. 31-33). London: Falmer Press. Ravetz, J. R. (1987). Usable Knowledge, Usable Ignorance; Incomplete science with policy implications. Knowledge: Creation, Diffusion, Utilization, 9(1), 87-116. Sanford, N. (1970). Whatever happened to action research? Journal of Social Issues, 26(4), 3-23. Schn, D. A. (1983). The Reflective Practitioner. New York: Basic Books. Schn, D. A. (1995). The new scholarship requires a new epistemology. Change, 27(6), 26-29. Schn, D. A. (Ed.). (1991). The Reflective Turn: Case Studies in and on educational practice. New York: Teachers College Press. Schwandt, T. A. (1996). Farewell to Criteriology. Qualitative Inquiry, 2(1), 58. Toulmin, S. (1996). Concluding Methodological Reflections: Elitism and democracy among the sciences. In S. Toulmin & B. Gustavsen (Eds.), Beyond theory: changing organizations through participation (pp. 203-226). Amsterdam: John Benjamins. Whitehead, A. J. (1997-present). ActionResearch.net. Whitehead, Jack. Available: http://www.bath.ac.uk/~edsajw/ [2000-present. Whitehead, A. J. (2003, 28 November 2003). Improving Student Learning through Passion in Our Professional Practice. Paper presented at the Annual Conference of Ontario Educational Reearch Council, Brantford.

Evaluation

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Evaluation

Hochgeladen von

Copyright:

Verfügbare Formate

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Appendix 11.1.1 Evaluation Thinking Experience Stories

Values in Acting, in Learning to Change

But What about the Values Bit? You Ask

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

8. 9. 10. 11. 12.

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Research and Evaluation Story 14 September 2001

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Contributing to Learning to Change (Drafting for thesis, 2004; edited out)

Das könnte Ihnen auch gefallen