1 Poster presented at 10th Biennial Conference of International Society for Theoretical Psychology (ISTP) in Istanbul on June, 24, 2003

RATING TASKS IN PSYCHOLOGY:

From construction of static ontology to dialogical synthesis of meaning
Brady Wagoner (Apollodionysian@aol.com) Jaan Valsiner (JValsiner@clarku.edu) Department of Psychology Clark University Worcester, Ma. 01610, USA

ABSTRACT. Psychologys methodology includes rating scales as quick and seemingly unproblematic method for data collection. We will demonstrate that rating scales are minimalist version of introspection, where easy quantification of the data obscures the psychological processes that generate the marks put onto the scale. Traditional rating scales intrinsically assume there is a single interpretational trajectory for subjects when they decide where to make a mark on the scale. This is possible only if we assume that we have an immediate, unitary, and accurate access to our own mental states, and that these states are inherently quantifiable. None of these assumptions is tenable. We look at rating tasks as constructions of subjective phenomenological space (SPS) which is accessible to the host (person) through meaning construction pathways (MCP). The rating task superimposes onto MCPs pre-set evaluative meanings and their scale-type quantifiers (e.g. Likert scales). This creates unitary (monological) construction of data out of the SPS, while the reality of mental construction processes that is involved in the rating is dialogical. Empirical examples of the study of rating process are given.

Rating scales are methods that have lost the possibility to provide the researcher direct access to psychological phenomena. This statement is obviously heretic (and meant to be so) in respect to the accepted practice of psychology where rating scales are an ordinary, easily administrable, andin their easy quantifiabilityeasily analyzable tool. We want to demonstrate that this ease is but an illusion, and that a radically re-organized look at rating scales from the perspective of the meaning-making processes that it triggers is in order.

The accepted empirical practice of using rating scales in psychology has shown the remarkable ease with which these scales can be applied. People when instructed to mark down their opinions on a graphic scaleseem to be able to rate almost anything under the sun? The rating scales may be created by way of juxtaposing opposite words, graphic images, even full picturesand linear, numeric, or verbal gradations of any kind can be established between the opposites. The research participants do not get confused when they are asked to fill out a rating scaletheir physiognomic relating with their world allows them instant (or at least uncomplicated) gestalt formation to provide quick answers. Sono doubt that by consensus of researchers and participants in research, rating scales are easy to use. Yet the ease of use may be an illusion of solidity of a method. We claim here that rating scalesdespite their easy usabilityare a complicated way to access psychological phenomena. In fact the use of such scales conceals rather than revealsthe psychological processes that are functional in the persons relating to the rating task. If we are interested in the access to psychological processes of making sense of the worldrather than mere outcomes of those processes, rating scales would misfit with the assumptions about the phenomena (Branco & Valsiner, 1997). Traditional rating scales intrinsically assume there is a single interpretational trajectory for subjects when they decide where to make a mark on the scale. This is possible only if we assume that we have an immediate, unitary, and accurate access to our own mental states, and that these states are inherently quantifiable. None of these assumptions are tenable. Even if human perceptual activity may be mostly immediate (as we learn from the Gibsonian, as well as physiognomic, perspectives (Werner, 1927), the making of sense entails construction of signs upon the physiognomic field (Bhler, 1934). That construction is usually automatizedhence the possibility of speedy ratingsyet the process of construction of meaning remains hidden behind the responding process. Theoretically speaking, rating scales are prosthetic devices that constrain the process of sense construction and enforce its speedy nature. As in any phenomena of reactionthere is a creative process of construction of the new sense involved. Rating scales trigger a microgenetic (in the sense of Sanders Aktualgenesesee Valsiner & van der Veer, 2000, chapter 7) process which is then speeded up to arrive at a conclusion. This focus is precisely the opposite of the need for slowing down the basic psychological processes in order to gain a glimpse into their organization. The argument for the latter goes as follows: most established psychological processes have become abbreviated (see Lyra, 1999) and automatizedand hence difficult to observe by the researcher. In order to gain a glimpse into their functioning, the natural speed of the processes needs to be slowed downby obstructing conditions.

3 What happens when rating scales are put to use? Any relation to the world entails making distinctions. Some of these distinctions to be made are guided by whoever sets up the task. Thus, the structure of the rating scale task constrains the structure of the answer and the way in which subjects are allowed to conceptualize the problem: the opposition of the end points of the scale is a given, the opposites are treated as mutually exclusive, and the scale intervals are assumed to be established and fixed. The minimal case. The first rating scale, technically speaking, is that of figure/ground distinction. A person is facing an empty field and is expected to report when s/he sees some figure in that field. Once that figure appearsby persons construction of such figure out of the empty surfacethe distinction made is the first (minimal) rating. That rating is not quantifiedit is a nominal scale point distinguishing the figure from the ground. Of course the researcher can ask the observer (to use the old-fashioned term for subject or research participantDashiell, 1929) to rate some features of the image. Through such instruction, the nominal scale entity becomes quantifiedalbeit in ways that are ambiguous as to their scale type (ordinal, interval or ratio). The primitive rating scale: forced choice. The rating system given to the observer constrains further the quantification process of the subjective experience. If they are given a dichotomoustrue or falseresponse format, the subjects are guided towards representing their experiences in black and white terms, excluding any middle ground. Thus, their interpretation is being guided by this response format structure. That subjects are involved in qualitative construction process that goes far beyond the superimposed response structure has been demonstrated well (Bhler, 1906; Valsiner, Diriwchter & Sauck, 2004). Yet as the forced-choice format requires streamlining of the process of responding into two channels, the access to that process is inadvertently lost: TASK ONSET TRUE

INITIAL COMPLEX FEELING

CANALYZING CONSTRAINT

FALSE

TIME

When given four check points on a scale to choose between subjects thoughts must be articulated into opposites, as well as not completely into the opposites pre-given-- but predominately black or predominately white. A full scale allows for many possibilities: Target: What is the color of this page? BLACK |--------|--------|--------|--------| WHITE This scale technically allows free movement of the rating processbut the markers offer a task demand to the rater, as if making their job easier. It is assumed that whatever verbal meanings may be projected onto the markers on the line scale (e.g., the mark to the right from BLACK is PREDOMINANTLY BLACK), the linearity of order between the two opposites prevails. The sense made through the linear scale is expected to represent the linear ordering of the raters experience domainthe subjective field to which there is no direct access to anybody but the person oneself. Even more importantlythe suggested responding format constrains the structuring of the subjective experience field. That field may consist of a dynamic flow between HATING and LOVINGwith a loop-like cyclical movement between the two states (see Figure 2, belowstate B). This state of affairs entails a case of unity of opposites: LOVING becomes HATING and the latter moves back to LOVING. The pre-given linear rating scale (Figure 1, state A) guides the person towards enforced linearization of the experience in the process of transforming phenomena into data (data derivationKindermann & Valsiner, 1989). The resulting hybrid (Figure 1 state C) is the subjective transformed state of the original phenomenon (B). Since it is transformed in the direction of the pre-given rating scale (A) it can be reasonably adequately mapped upon the linear scale (D). Yet it is obvious that there are no guarantees of the isomorphic mapping of C onto D (that could happen only iff C has become completely linearized) hence the resulting linear scale is at most guaranteed the status of the ordinal scale. In Figure 1, the linearized subjective experience (C) has also eliminated the dynamic process aspect of the original phenomenon (B)especially in the global crossing zone between LOVING and HATING and vice versa. That zone is the area of dynamic ambivalence. That dynamics of ambivalence has become inadvertently lost from the linear rating scale. Yet the ambivalence may be recorded on the linear state as an outcome. Both sides of the scale may feel like fitting at some time, under some conditions. Yet as the instruction requires one single marker to be put onto the scalethe traces of the complementary dynamic process of LOVING feeding into HATING (and vice versa) become inadvertently lost through enforcement of the rating operation.

(A) SUGGESTED RESPONDING FORMAT HATE --

(B) CURRENT SUBJECTIVE EXPERIENCE

SUPERIMPOSED

HATING

LOVING

-LOVE LEADS TO BREAKING THE ACTUAL SYSTEM (C) TRANSFORMED PHENOMENON

HATING

LOVING

HATE

LOVE

(D) RESULTING MAPPING OF TRANSFORMED PHENOMENON ON A LINEAR SCALE

FIGURE 1. The assumed process of linearization of dynamic and systemic subjective experience by introduction of a linear rating scale The paradox of elimination of the phenomena through enforced rating is better illustrated in Figure 2. Rating Figure 2 on a scale between grove shelf and tongue shelf would likely result in a middle rating. There are two contradictory perspectives that we can take toward the object: It either represents a grove shelf or a tongue shelf, but our rating of the object in the middle involves a dynamic interplay between the two. The object is only contradictory at the meta-level analysis, or were we force the subject to make a choice between A or B. Subjects are unable to express that the object is both A and B through a linearized rating. The linearization of the interpretation of the object assumes

6 that our interpretation is stable that there is no perspective taking but merely a single quantifiable perspective on the object. Our minds can move quite naturally between the two perspectives, through a transition point were the object is neither A or B, and/or is both A and B in the transition zone.

Tongue Shelf<<<<Transition Zone>>>>>Grove Shelf Figure 2 Fluidity of Rating and the transition between perspectives. This confounds the Cartesian claim, which the traditional rating scale assumesthat ideas are clear and distinct. In fact, it is precisely the ambiguity at the edge of our conceptshere magnified by the visually ambiguous figure-that allows us to expand on our ideas, allowing movement from one perspective to another. The problem of the middle point: theoretical varieties These examples should demonstrate that the use of linear ratings scales in case of systemic phenomena involving transitions creates data that do not represent the phenomena any more. Yet even in cases where such distortion is not happeningwhere we deal with linear processes aligned as vectors on the given scale, the meaning of one or another mark on the scale is ambiguous. If the researcher assumes that there is a choice between two essences (opposites) as to their fit with the rating object (the entity assumption), the middle rating either entails equal quantity of both, or impossibility to decide. It is the researchers background assumptionsnot the empirical datathat enforce this interpretation upon the middle point. The meaning of the rating-in-the-middle would be very different if the researcher begins from an assumption of any rating being a result of dynamic and opposed forces. Then, an X in the middle of a linear scale represents the resultant of two opposite forces in some status quo. Two oppositional meanings with equal pulling force create a seemingly neutral rating. In the example of this (given below), the female subject was asked to rate and explain her rating of a picture of a male on the category approachable.

Not at all

Approachable |---------|---------X---------|---------| <Object>

Very much

In the inquiry into the subjects rationale of putting the marker in the middle, we get: Personally approaching guys especially nice looking guys is not a common practice so I wouldnt deem him all that approachable but he would probably be used to girls approaching him so it evens out. The subject begins to understand and rate the object by imagining herself as approaching the male. This naturally flows into imagery supportive of the opposite rating, namely lots of girls approaching him, which means he must be approachable on some level, or at least with the approaches she imagines are taking place. The process of answering entails the construction of dynamic forces projected to the qualities of the end points of the scale. It is not only the meaning of the object of rating, but also of the system of rating, that the respondent constructs when put into the rating task setting. Example 1 (from a study of inter-gender relations) In a study on the dynamics of approach and avoidance in inter-gender relationships, Wagoner (2002) had subjects make several scale ratings of a pictured person. Subjects were then asked to write a letter to the pictured person expressing their interest. Some examples from the first part of this study will illuminate the dynamic interplay of opposites that takes place in rating. All these scales described thus far require the subject to articulate his or her rating as a static characterization of the rated object. Subjects juggle several interpretations mutually influencing and developing each other to finally come to a static completion of the task. The nature of this task requires that subjects understand (to rate) the object as a static category. But it is not a static rating. Ratings are the product of oppositional forces (voices) pulling toward the ends of the scale (with the rare exception of monological ratings). In the process of linearization, the dynamic flow of opposites is transformed into pulling forces on the rating scale. In the following example (below), the female subject was asked to rate a picture of an attractive male on the category of courteous, and explain her rating:

Courteous |---------|---------|----X---|---------| Very much <Object> I can see this guy holding doors for the ladies and buying flowers for his girl, etc, which is courteous/nice but I can see him having multiple girls. Not at all The oppositional pulling forces (argumentative voices) create a tension that leads the subject to make a mark in almost the middle of the scale. Holding doors and buying flowers for girls is a courteous gesture, although these images lead the subject to imagine another possible scenario in which the pictured male is allotted the respondents antipathy (having multiple girls). It is important to note how the opposite meanings are related to one another, how they flow into one anotherthe later, but, sentence is an extension of the sentence before itit argues from the opposite side of the scale. The abstract concept of courteous is ambiguously related to various scenarios; it takes on sensuous imagery as well as becomes linked with moralistic discourse.

Alternatives to linear rating scales Alternatively, a mark in the middle of a linear scale may represent an emergent meaning, going beyond the given dichotomyin effect, a rejection of the dichotomy. Figure 2 (above) illustrates a case where the dichotomy is rejectable. Rating a male on the dichotomous scale FEMININE |---------|---------X---------|---------| MASCULINE and assuming the unity of the two opposites (i.e., androgyny) leads to the emergence of the latter meaning as a synthesis of both masculinity and femininity: FEMININE |---------|---------X---------|---------| MASCULINE

ANDROGYNOUS

9 New Rating Scales To capture the dynamic nature of rating and avoid a forced linearization upon the rating subject we developed two novel rating scales. The first is an expansion upon the traditional rating scales, which provide a final product but do not show how subjects negotiate this producttraditional scales leave out the process. Subjects were instructed to rate as they normally would but add other possible ratings for the object. This was accomplished by having subjects draw arrows from the X that they mark, and ask them to justify their response in writing. For example, a rating on the traditional scales for the object below would look something like this: Black |--------------------X-------| White Our new scales would look something like this: Black |---------------------X-------| White Why: The largest part of this page is white but there is still quite a bit of black. In fact, one could argue this is the part of the page people pay attention to and therefore the essence of the page. However, when I think of a page I image whiteness not black black is secondary. We can then schematize the dialogue as a movement between the two perspectives:

Black |----------------------X------| White <<<Object>>>>> The largest part of this page is white but there is still quite a bit of black. In fact, one could argue this is the part of the page people pay attention to and therefore the essence of the page However, when I think of a page I imagine whiteness not black black is the secondary association.

10 The dynamic dialogue when articulated onto the scale becomes organized in the form of oppositional pulling forces, which compete for where the mark is to be made on the scale. The set of arrows above represent the negation of the marking through the pulling ends. The line running in between the dialogue represents a possible trajectory of movement within and between the arrows marked above. The decision to make the mark toward the white end is a result of the greater pull toward the white side (the tension is represented with the <<>> and <object>). This scale has been very useful capturing the interplay between the two pulls, however it does not allow subjects to express their uncertainty/ambivalence about their rating outside the interchange of voices and arrows. Often, subjects reject the task altogether when making a rating in the middle of the scale. To account for the emergence of new meaning, in the case of a rejection of the task of rating, we created a three-point scale (see below). Triangular rating scales Still, the linear scale would not allow for the distinction of the simple balance of power of two opposite vectors and the synthesis of a systemic unity of the two. A scale suggested by Alexander Liebrucks can allow for such distinction: ANDROGYNOUS

FEMININE

MASCULINE

The rater is expected to mark the area within the triangle (as well as put down a centering mark within it). If all the marks are made on the base of the triangle, the rating task becomes equivalent to a linear scale. Hence, the traditional rating scales constitute a special case of the Liebrucks scale. Consider an examplea person is asked to rate something (we can leave open the precise nature of that something) on the following scale:

Unknown

Good

Evil

11

Three-pulling forces are involved in the above scale, which allows subjects greater freedom in their ratings. That is, they are able to make novel interpretations against the forced dichotomy of good and evil.

Though really it is nothing more than marks on a page. How could that be either good or evil? 1) The figure appears neither good nor evil to me

Unknown

Good

Evil

2) Though when I look at it more closely does give me the chills 3) But good art and images should have a stirring effect on the subject, so in that sense it is quite good 4) Still the picture itself is in away sinister, dark and in a word evil This elaboration shows how this scale acknowledges sites of ambivalencein the case of the above rating the ambivalence is directly between the two opposites. The graph could just as well point to ambivalence while making judgments at either end of the Good/Evil dimension. Example 2. Results from the use of triangular rating scales. A study with university students in Estonia (N=45) was carried out, asking them first to describe what they see in a particular image and what feelings and associations that image provokes. One of the images used was the following:

12 After the subjects had described this (and two other) graphic visual stimuli, the triangular rating scale was introduced (including an example). This was followed by use of the rating scale in relation to the already described objects. The following three examples from these data provide evidence for the ways in which the meaning construction process proceeded. Subject # 12 (21 year old Estonian woman) Description: This is a long corridor to an unknown land, place. Like from a Bond movie (at the beginning of a movie, there is something similar) It evokes interest, excitement, like something very interesting is coming up. Rating:

Explanation: Doesnt look bad and hurting. I want to enter it but dont know what will be waiting for me there.

Subject # 29 (24 year old Finnish woman) Description: Some kind of a tablecloth from the seventies, with a psychodelic pattern. Makes you dizzy if you look at it too long. It also reminds me of a movie I recently saw about a woman who got to visit spacethe picture looks like a wormhole like they were described in the movie. Rating:

13

Explanation: There is something familiar to it, but still something I wouldnt want to be faced with. Subject # 34 (22 year old Estonian woman): Mingi mstiline vli, mis imeb endasse. Justkui neelaks endasse. rev tunne.[Some mystical field, that sucks into itself. As if swallows into itself. Anxious feeling] Rating:

Explanation: Segu tundmatust ja kurjusest-elusest. Seal sgavikus vib oht varitseda, aga ma ei tea seda [Mixture of unknown and evil-meanness. In that depth there may be a danger, but I do not know it] As is obvious from these examples, the triangular rating scale made it possible the expression of the uncertainty about the subjectively constructed meanings of the objects (that meaning construction had already taken place in phase 1 of the study, before the rating tasks were introduced. Although the task made it perfectly possible to translate the triangular scale space into the unidimensional linear scale (i.e., a simple GOODEVIL linear scale), only 4 out of 45 subjects did it (the designated area merged with base line, with X marker touching the base line). In addition, only 2 subjects used the minimum possible area (pointindicated only by X- mark, without designating a bounded sub-field) in their rating of this object. When given an opportunity to express the range of subjective meaning of the objectin this case a purely geometric pattern subjects in their majority follow that opportunity (rather than translate their estimates into point-like linear scale marks). The examplesempirical and possibleindicate that the rating process is a complex meaning construction task. The following model is an effort to reconstruct what happens in that process.

14 Access to Subjective Phenomenological Space (SPS) The meaning-making process in the case of rating is of the kind of creating a field of meaning (A) with its legitimate opposite field (non-Asee Josephs, Valsiner & Surgan, 1999). The actual rating is generated through the tension between the A and non-A fields. In the above examples, the subject gets a feeling from the object of rating, which we have referred to in the abstract as subject phenomenological space (SPS). |-------|---X---|-------|-------|

MCP

SPS

FIGURE 3. The Subjective Phenomenological Space (SPS) and Meaning Construction Pathways (MCP) Put differently, SPS is the constant flood of every changing sensation that we are implicitly aware of. Thus, this is not enough to make a rating of the object. If it were, subjects would do so immediately and without any conscious thought. To make any sense of the SPS people must bridge it with abstract concepts that capture the fluid sensations of the SPS in a form. This form directs, refines, and controls the imagery that comes to mind and the narrative that follows toward the form. The instrument by which we understand the SPS and control it is the meaning construction pathways (MCP). Several MCP can access the same SPS, each creating different images and voices. The subject in the above example uses two MCP for the form courteous to understand the picture presented to her (see Example 1, above).

15 General Conclusions Any interpretation of an object to be rated is in a constant flux. The rating process involves balancing this inherent instabilityor rather stability in instability, the mark of dynamic systems. Our concepts are fuzzy at the boundaries, which allow the movement between perspectives. There is a unitary movement between the two poles of the scale (the unity of opposites) that creates a pull between the two pulls, in contest over where the definitive rating mark will be made. Scales that capture the process of rating need to be created to properly understand the meaning of the rating. Ratings in the middle of the scale cannot be easily quantified because of their ambiguous nature, living between the interchange of two perspectivesat the boundary of both. Instead of throwing out these uncertain subjects we should rejoice in their existence and the keys they hold to unlocking deep clues into the mysteries of our psychic life. New kinds of rating tasks can be developed that can open the door to investigation of the processes of meaning construction. These scales may take unusual geometric formsyet psychology should be ready to get out of its illusion that simple assignment of numbers to complex processes equals measurement (see Michell, 2003). Our methods are created on the basis of the complete methodology cycle (Branco & Valsiner, 1997) that requires preservation of the selected crucial features of the phenomena in the constructed data. That principleof representativeness of the phenomena by the data is not honored in the majority of the uses of linear rating scales. Psychologists using traditional rating scales may be producing artificialyet consensually accepteddata that is an obstacle for further scientific progress, rather than its basis. Acknowledgments. The work on this paper was supported by the Howard Jefferson Prize awarded to the first author by Clark University. The presentation of the paper at the conference was made possible by travel funding from the Hiatt Fund of Clark Universitys Psychology Department.

References Branco, A. U., & Valsiner, J. (1997). Changing methodologies: A co-constructivist study of goal orientations in social interactions. Psychology and Developing Societies, 9, 1, 35-64. Bhler, K. (1907). Tatsachen und Probleme zu eine Psychologie der Denkvorgnge. I. Archiv fr die gesamte Psychologie, 9, 297-365. Bhler, K. (1934). Sprachtheorie. Jena: Gustav Fischer. Bhler, K. (1990). Theory of language: The representational function of language. Amsterdam: John Benjamins.

16 Dashiell, J. F. (1929). Note on the use of the term observer. Psychological Review, 36, 550-551. Josephs, I. E., Valsiner, J., & Surgan, S. E. (1999). The process of meaning construction. In J. Brandtsttdter and R. M. Lerner (Eds.), Action & self development (pp. 257-282). Thousand Oaks, Ca.: Sage. Kindermann, T. & Valsiner, J. (1989). Strategies for empirical research in context-inclusive developmental psychology. In J. Valsiner (Ed.), Cultural context and child development (pp. 13-50). Toronto-Gttingen-Bern: C. J. Hogrefe and H. Huber. Lyra, M.C.D.P. (1999). An excursion into the dynamics of dialogue: Elaborations upon The Dialogical Self. Culture & Psychology, 5(4): 477-489. Michell, J. (2003). The quantitative imperative: positivism, nave realism, and the place of qualitative methods in psychology. Theory & Psychology, 13, 1, 5-31. Valsiner, J., & van der Veer, R. (2000). The social mind: Construction of the idea. New York: Cambridge University Press Valsiner, J., Diriwchter, R., & Sauck, C. (in press, 2004) Diversity in Unity: Standard questions and non-standard interpretations In R. Bibace, J. Laird, K. Noller & J. Valsiner (Eds), Science and medicine in dialogue. Stamford, Ct.: Greenwood. Werner, H. (1927). ber Pysiognomische Wahrnehmungsweisen und Ihre experimentelle Prfung. In Proceedings and papers of the 8th International Congress of Psychology, 1926, Groningen (pp. 443-446). Groningen: P. Noordhoff.