A Framed PAINTING:

The Representation of a Common Sense

Knowledge Fragment*
EUGENE CHARNIAK

University of Geneva
This paper presents a "frame" representation for common sense knowledge and uses it
to formalize our knowledge of "mundane" painting (walls, not portraits). These frames,
while designed to aid a computer program to understand stories about the painting process,
should be of use to programs which attempt to actually carry out the activity. The paper
stresses a "deep" understanding of the activity so that the representation indicates not Only
what steps to carry out, but also how to do them, and why they should be done. To
accomplish this, while at the same time preserving modularity and nonredundancy, a
system of interframe pointers is introduced (the COMES-FROM and LEADS-TO pointers) which explain how or why something is done in terms of knowledge given in other
frames. The paper proceeds by steadily deepening an initial English-like description of the
activity, and a context free grammar for the representation is included.

There is a c o m m o n agreement among Artificial Intelligence researchers that

intelligent behavior requires large bodies o f domain-specific knowledge, be it
chess, medicine, or the general " c o m m o n s e n s e " knowledge which all m e m b e r s
of our culture share. Naturally, such agreement has led to a rapidly growing
literature about how such knowledge could be represented and organized. The
present paper is an addition to this literature.
In particular I will develop here a formalization o f our k n o w l e d g e concerning
what might be called " m u n d a n e " painting. That is, we will be concerned with
our knowledge of how one goes about painting objects like walls and bookcases,
as opposed to the " a r t i s t i c " painting o f still lives and portraits. I am currently
designing a program which will use this knowledge base to understand simple
stories about painting and demonstrate its understanding by its ability to answer
questions about the stories (see Charniak, 1977). But while I shall be primarily
concerned with the use o f such knowledge in language comprehension, the
forfiaalism developed here should be equally relevant to the construction o f
programs which are able, in an intelligent fashion, to actually paint. That the
representation have this property is one o f the constraints I have adopted, on the
grounds that a special and distinct knowledge base, used solely for language
*Reprints of this article may be obtained from Professor Eugene Chamiak, Yale University, Department of Computer Science, 10 Hillhouse Avenue, New Haven, Conn. 06520.
355

356

EUGENE CHARNIAK

comprehension, seems quite implausible, at least for activities like painting. This
too is a common view in Artificial Intelligence circles, but little if anything in the
present paper depends on it. In particular, while upon some occasions I will
introduce a new fact into the representation because it is clear that a real painter
would need such a fact, in all cases an appropriate story could be constructed to
demonstrate the same need in the domain of question answering. Indeed, for the
less intuitive cases I will provide such story examples.
The consensus mentioned in the opening paragraph allows one to safely skip
over the otherwise obligatory justification for the line of research in question ,,But
in doing so, it simultaneously presents a different and, if anything, more difficult
task--that of distinguishing one's research from the many papers on clearly
related topics. To this end let me note that the representation presented here has,
at its heart, the problem of depth of understanding and the concurrent complexity
of knowledge which reasonably deep understanding involves. So, in the domain
of painting, when I speak of depth of understanding, I mean that it is not
sufficient for a program to know that one should wash a paint brush after
painting--one should also know how to do it, what will happen if it is not done,
and why the painter's intention to throw the brush away after using it constitutes
a reason why the otherwise useful suggestion can be ignored in this case. Similarly, if we wished to formalize our knowledge of how to send a letter, we should
indicate not only that a stamp should be affixed, but again, how, why, and what
may happen if it is not done.
To see how such deep understanding involves us in problems of complexity of
related knowledge, let us consider the stamp example in a bit more detail. A first
guess at how we might organize our knowledge of letter mailing would be to
write down a series of instructions which explain exactly what one does in order
to accomplish the task. This is especially reasonable given the earlier comment
about wanting the representation to be applicable, not only to story comprehension, but to actually performing the activity. But as soon as we decide to organize
our letter mailing frame in this fashion we are immediately confronted by the fact
that the reasons for doing many of the activities therein have nothing to do with
the actions one is performing, and everything to do with the actions one expects
the post office to perform. Indeed, having once decided on a letter mailing frame,
one immediately needs a post office frame, and complex interconnections between the two; something which might look like Fig. 1.
One might, of course, simply dispense with two separate frames, and combine
the two into one, but there are several reasons against such a maneuver. For one
thing it would become more difficult to see our frames as telling us how to
perform the activities in question, since they would now include many actions to
be performed by others. Furthermore, one reason for splitting up our knowledge
in the first place is to help beat the combinatorial explosion. That is, we want to
keep down the amount of information we need at any point of a story, or at any
point in our real life activities, since failure to do so means rapidly growing

A FRAMED PAINTING

FIG. I

357

The relation of letter mailing to the Post.

problems of searching our knowledge base for those facts which are indeed
relevant to the situation at hand. Finally, although this is not so obvious in the
above example, in many cases (including those we will encounter in painting) to
effect such a combination requires repeating pieces of knowledge many times
over because they come into play in many situations. Given the enormous numbers of facts people seem to use, to multiply our storage problem by 50 or 100 by
such profligacy seems inadvisable at best.
If we turn now to painting we note that it is normally performed by a single
person. Hence we do not have exactly the kind of interdependence between letter
mailing and post offices. But we do have something similar in the dependence of
painting on the common sense physics and chemistry that underlie it. For example, if one is to understand why we wash a paint brush we must understand the
general properties of evaporation, the specific properties of evaporated paint, and
the advantages of having a paint brush that is both flexible and absorbant.
Washing in turn requires knowledge of mixing liquids, the nature of
homogeneity, and how liquid sticks to surfaces. It would be redundant to include
all this information directly in PAINTING, so we are again faced with the
problem of relating the complex event, PAINTING, to other events on which it
depends.
Requiring that the representation be capable of expressing such complex interrelated knowledge has led me to illustrate the representation by the constant
appeal to one topic, namely, mundane painting. It is only in this way that I can
build the needed knowledge of painting and its related' topics, which together will
be referred to as the painting complex (see Fig. 2). In taking this approach, of
course, I risk the possibility that the representation presented will not extend
easily to other domains. But to do otherwise, and cover a wide variety of subjects
would effectively bar the depth I seek.

358

EUGENE CHARNIAK
rollerpainting

/ I~

/
absorption,

p.......

fa:l

/~.~"~-~'~\
/

~/

resid . . . . . l i d i fication

~/

vol(ume)-inc/(teasel / ~

:"
halt

J S
fluid-containment

//

~'~

~
co~ino

residue Lcontac//t I ~

......'

....

II \stuff-color
homogeneous

sticky-on stick separatecontact part-of change support exteriorsurface

FIG. 2 The better developed frames in the painting complex. The unconnected frames at the
bottom play a role in so many frames as to prohibit linking them to the frames in which they appear.
For the most part, the rest of this paper describes in steadily mounting detail
the painting complex and the representation in which it is written. The exception
is Section 3, the conclusion, which is a fairly general discussion of future work
and the nature of semantic representation. Section 1 gives the broad outlines of
the representation, and in particular introduces the C O M E S - F R O M and
L E A D S - T O pointers which are used to effect linkages like those we saw in the
letter example. Section 2 will describe the representation all over again, but now
we will be much more concerned with the details like the representation of
continuous actions, and what sorts of variables we need in our frames. A still
greater level of detail, the entire painting complex (see Fig. 2) expressed in full
formalism and freely commented, has been omitted from the paper for reasons of
space. It is obtainable in an appendix to a slightly longer version of this paper
(from the Institute for Semantic and Cognitive Studies) or separately from the
author.
I.

1.1

THE FUNDAMENTALS OF THE REPRESENTATION

Frames in General and Painting in Particular

The reader may have noted that although I have used the word " f r a m e " two or
three times, I have yet to define it. In fact, I have been trading on the current
popularity of the term and assuming that the reader will have a general idea of
what is meant by it. But for the sake of completeness, let me try to give a
minimal definition in the sense of one which will include all current uses, but
which is no doubt insufficient to discriminate some nonframes from frames.

A FRAMED PAINTING

359

I take a frame to be a "large" body

of information

(as opposed to a single predicate

calculus statement)

expressed in a computer usable

formalism

(and in particular not in natural

language)

about a single topic

(as opposed to a formalized

encyclopedia)

where the facts are richly

interconnected

(unlike a computer listing of stock

market prices)

and which, while typically consulted

as a whole

(as opposed to the usual sort of

semantic net)

consists of basic units (facts) which are

individually useful

(as opposed to a single statement in a

matrix multiplication routine).

I orginally stated that I expected my painting frame to be useful not only in

understanding stories, but also to contain information that would be used to guide
the actual act of painting. To satisfy this second criterion requires, at the very
least, that the frame specify what things need to be done, and in what sequence.
That is, our painting frame might look something like Fig. 3.
In Fig. 3, I have adopted a convention, which will be used throughout this
paper, of capitalizing those portions of the frame which are identical (or nearly
so) to the final version mentioned earlier. This is why, for example, the statement names, PAINT1, etc., do not have contiguous numbers. (It also gives
some idea of how much is left out of this first try.)
The portions in lower case are simply English phrases to stand in for the
formalism which, by the end of the paper, will replace them. The symbol G O A L
marks the typical goal state of the event being described, while E V E N T marks a
more detailed description of the activity itself. The arrows, of course, simply
indicate the time ordering of the various steps. Lastly, note that I have made the
frame neutral between using a brush or roller by mentioning neither and instead
using the term "painting instrument." Otherwise I would think that the example
should be reasonably self-explanatory.
PAINTING try number 1
GOAL : PAINTING-GOAL (object to be painted has a coat of paint on it)
EVENT: FPAINTINGI4 (object is not dirty)
~PAINTINGI6 (LOOP
PAINTINGI7 (get paint on the painting instrument)
PAINTING22 (bring instrument in contact with object)
)
LPAINTING37 (all paint removedfrom painting instrument)
FIG. 3 PAINTING try number I.

360

EUGENE

CHARNIAK

Actually to use such a knowledge representation to paint would be no easy

matter. In some sense we would want to consider P A I N T I N G a pi'ogram and
then proceed to "execute" it. But precisely how this would be done is not clear.
In the present case this is complicated by the obvious sketchiness of P A I N T I N G
try I, but even were we in possession of a complete P A I N T I N G there would still
be much necessary information left out. In particular, to actually paint would
require basic knowledge about vision, or how to move one's arms and legs, and
clearly neither of those topics would be represented in P A I N T I N G . Before we
can make much sense of the notion of "executing" P A I N T I N G , the relation
between this level of knowledge and those more basic levels underneath must be
explicated. However, this is no task to be taken up as a side issue, as it would be
here.
The information expressed in P A I N T I N G , however, is also required for
answering questions about stories concerning painting. For example:
Jack was going to paint a chair. He washed the chair.
(Question: Why? Answer: One should not paint a dirty object. Q: Why
not? A: It can lead to flaking.)
Then he wiped the handle of his paint brush.
(Q: Why? A: I don't know.)
A little bit later Jack dipped the brush in the paint.
(Q: Has he finished yet? A: No.)
Then he drew the brush across the chair.
(Q: Could this step be left out? A: No. Q: Why not? A: Because it is this
step which gets the paint on the chair.)
Some time later he started to clean the brush.
(Q: Could this step be left out? A: Yes. Q: Then why is he doing it? A:
To prevent the paint from drying on the brush. Q: Is that bad? A: Yes
because it can make the brush unabsorbant and inflexible which would
make it useless for painting.)
" I will use some newspaper" thought Jack:.
(Q: For what? A: Probably in order to wipe the brush on it.)
Besides showing how such knowledge can come into play in story comprehension, this example also illustrates the level of understanding which the representation developed here should be capable of exhibiting. As literature, such examples are obviously quite unrealistic, but it seems clear that if we cannot get a
program to understand such examples, we will have little hope for more complicated (and realistic) ones.
To use P A I N T I N G to understand stories about painting, while by no means
trivial, at least seems much more tractable than using it to paint. We will not

A FRAMED PAINTING

361

discuss the details here (primarily because they have not been worked out) but the
basic idea is quite simple. (The reader might consult Charniak (1976) for a
longer exposition of the following ideas. For that matter he could also consult
Schank et al. (1975) or Rieger (1975) where much the same ideas appear.)
Essentially the input sentence is seen as instantiating one or more lines of one or
more frames. That is, when we see Jack washed the chair in our earlier example
the program is to understand this as instantiating P A I N T I N G I 4 (object not
dirty). It is the role of the parsing algorithm to translate the sentence into a form
which can later be used to make such identification between what we are told in
the story and what we have stored in our frames. Once this identification is
made, then inferences, such as how far along Jack is in the act of painting, or
why he did something, are to be made by referencing the frame. Obviously, if
washing the object appears as it does at the beginning of the event time sequence,
then Jack cannot have finished his activity.
To flesh this out, we must specify how parsing works, how we select appropriate frames, and how we go about looking for matching statements within those
frames. But as I said, these issues will not be dealt with here.

1.2

Methods and COMES-FROM

We noted earlier that our first try at a painting frame contained nothing like the
necessary breadth of information. But equally important, neither does it contain
the necessary depth. That is, to repeat my earlier example, it states that the
instrument used in the painting should be cleaned, but not how and why. This
first portion of the paper will be primarily concerned with how such information
is to be included in the representation. We will start with the " h o w " type
questions.
By a method subroutine I mean one which in effect says, " y o u want to bring
about such and such state of affairs, well, I suggest using the following method."
There is a standard way in which such method subroutines are indicated, as for
example in P L A N N E R .
(GOAL

(some state of affairs)

(USE method-name))

Here method-name is the name of some P L A N N E R theorem which can be used

to bring about the desired state of affairs. Furthermore, it will have a theorem
pattern which matches (in the typical pattern matching sense) the goal pattern in
the. above G O A L statement. Adopting this notation to our slightly different
syntax, we might say:
PAINTING37

(all the paint removed from instrument)

use washing-frame

Indeed, with two small changes this is what I will do. The first change is minor,
simply being the replacement of the word " u s e " by the term C O M E S - F R O M .

362

EUGENE CHARNIAK

In the above situation this makes no difference, although my replacement is

admittedly somewhat less mnemonic. (One can think of it however as indicating
where the desired state of affairs C O M E S - F R O M , or equivalently, how it is to
come about.) However, there are other circumstances in which the word " u s e "
has undesirable connotations which C O M E S - F R O M does not share. In particular, " u s e " has the connotation of an animate agent who desires a particular
outcome. In many cases we will want to express how an often undesirable state
of affairs is the result of some purely physical process. In such a case we could
say that the state of affairs C O M E S - F R O M the physical process, but not that we
" u s e " the process to bring about the state of affairs. (Of course, from a purely
technical point of view the computer could not care less which term we used.)
One such example is involved in the complete specification of a rule we have not
yet included in the painting frame:
PAINTING15

(cover everything with paper)

Here "everything" only includes things we do not want spotted with paint.
Eventually, when we completely specify this rule, we will state that the spotting
C O M E S - F R O M the processes of D R I P P I N G and S P L A S H I N G . To say we
" u s e " these processes to create spots, in this context, would be a bit bizarre.
The second change to be made in the P L A N N E R type formalism is more
complex than replacing one word by another. As I mentioned earlier, there is the
convention in P L A N N E R that the pattern of the theorem must match the goal
pattern. In the course of making this match one binds the variables in the theorem
pattern so they reflect the special state of affairs mentioned in the goal. That is, if
our goal is to put block A on block B, our theorem will give general instructions
for putting one block on another. To specify this to the current situation, we
simply bind the appropriate variables in the pattern to "block A " and "block
B , " respectively. Given that there is only one pattern per theorem, by specifying
the theorem name in the " u s e " statement we automatically are specifying the
pattern to be matched. (Of course, one can call a theorem in P L A N N E R entirely
on the basis of the pattern without any mention of the theorem name at all. But
this is besides the point in the current discussion.)
In the representation presented here there is no unique theorem pattern in the
sense required by the P L A N N E R mechanism. Instead there may be many statements in a frame which serve as patterns. (Illustrating this would take us too far
afield, but to the reader interested in returning to this point, note (see Fig. 4 and
7) that in STICK, STICK6, 8, and 10 serve as "patterns" for P A I N T I N G 2 2 ,
PAINTING26, and PAINTING17, respectively.) The point here is that if we
are to give up a special and unique pattern, we must indicate what is to be
matched by some other means.
A bit of terminology would be useful in this regard. "Frame statements," or
simply "statements," are those things, like PAINTING37, that we use to build
up our frames. They invariably have a predicate plus argument structure, thus far

A FRAMED PAINTING

363

presented by English phrases in parentheses, "(all the paint removed from instrument)," which we will call the " b o d y " of the frame statement. The statement may also have other parts, such as a name, P A I N T I N G 3 7 , or tags like
" C O M E S - F R O M " which we are currently in the process of introducing. Given
this terminology, our C O M E S - F R O M tag must somehow indicate the frame
statement body which is to serve as the matching pattern. I will indicate the
appropriate frame statement in the most obvious way, by giving its name. (As I
just did here, I shall frequently refer to a frame statement body as simply a frame
statement in which no confusion will result.)
With the two changes described above, our representation for P A I N T I N G 3 7
becomes:
PAINTING37

(all paint removed from instrument)

COMES-FROM: (WASH-GOAL)

Here W A S H - G O A L is the name of the particular statement in the W A S H

frame. Its body, of course, must match the body of PAINTING37.
To give another example of the same construction, we must have a rule in
P A I N T I N G to the effect that one should not have too much paint on the painting
instrument, and that a good way to prevent this is to remove excess liquid by
pressing the instrument against a solid surface, causing some liquid to drip off.
We will call this procedure for the removal of liquids R E M - L I Q a n d our addition
to P A I N T I N G will be:
PAINTINGI9

(not too much paint on instrument)

COMES-FROM: (REM-LIQ-GOAL)

Let us give one last example of the use of C O M E S - F R O M , this one of a

different sort. So far we have used our C O M E S - F R O M pointers to indicate a
complex activity, like washing or removing liquids from an absorbant material.
We can also use it to indicate the physical process which underlies these activities. So we are told in P A I N T I N G I7 to get paint on the painting instrument,
but so far we have not indicated the physical process which allows us to do so.
This physical process, which we will call STICK, has its own frame, which as a
first approximation might look like Fig. 4.
In effect this says that if a solid comes in contact with sufficient liquid some of

Stick try 1
EVENT : I--STICK6(liquid in contact with object)
|WITH-OPTIONAL-CONDITIONS
I(STICK8 (amount of liquid is greater than some constant) )
[CAUSES
~'IF STICK8 THEN STICKI0 (liquid sticks to object)
ELSE STICKI I (liquid sticks to part of object)
FIG. 4 STICK try I.

364

EUGENE CHARNIAK

the liquid will stick to .the solid. The O P T I O N A L - C O N D I T I O N S are not

prerequisites of the event, but may affect the end result.-In this case if there is
less liquid only part of the solid will be covered. We can relate this physical
mechanism to P A I N T I N G I 7 in the following way:
PAINTINGI7

(get paint on the painting instrument)

C O M E S - F R O M : (STICKI0)

(Again, the bodies of S T I C K I 0 and PAINTING17 must match. The current

English expressions do not, but this will be ironed out when we move into the
complete formalism.)
With the changes suggested in this section, our P A I N T I N G now looks like
Fig. 5.
1.3

Reasons and LEADS-TO

In my original comment about wanting a representation which encourages a

deep understanding of the topic I suggested that it was not sufficient that the
program know that the brush or whatever should be washed--it must also know
how to do it, and why it should be done. In the last section we considered the
former aspect of the program, now let us turn to the latter.
In Charniak (1975) I commented that much of the information we have about
activities is in the form of "suggestions" rather than strictly necessary activities.
I noted there that each suggestion had a "reason" associated with it which
explained why the suggestion should be followed. Washing a brush (PAINTING37) is a clear-cut example, and the reason is simply that otherwise the
brush will become stiff and unabsorbant. I am no longer convinced that there is a
clear-cut class of "suggestions," for on close inspection they tend to merge with
strict requirements. But the notion that some frame statements should have
reasons associated with them seems unassailable to me, and it is this aspect
which will be discussed here.

PAINTING try 2
GOAL : PAINTING-GOAL (object to be paint~ has a coat of paint on it)
EVENT: PAINTING14 (object is not dirty)
PAINTINGI5 (everything covered with paper)
PAINTINGI6 (LOOP
PAINTINGI7 (get paint on the painting instrument)
COMES-FROM: (STICKI0)
PAINT1NGI9 (not too much paint on instrument)
COMES-FROM: (REM-LIQ-GOAL)
PAINTING22 (bring instrument in contact with object)

)
PAINTING37
FIG. 5

PAINTING try 2.

(all paint removed from painting instrument)

COMES-FROM: (WASH-GOAL)

A FRAMED PAINTING

365

Of course, P A I N T I N G 3 7 (all paint removed from instrument) is not the only

frame statement in P A I N T I N G that could use an associated reason. Two others
are:
PAINTINGI4
PAINTING19

(object is not dirty)

reason: otherwise paint may flake and crack.
(not too much paint on instrument)
reason: otherwise the paint will drip.

Now representing the suggestions is no problem. We have already seen that they
can be handled much like any other flame statement. The difficulty comes in
handling the reasons for the suggestions.
One possibility would be simply to include these reason statements in the
representation, much like I did above, althoug h naturally not in English sentences. This, however, is insufficient by itself, because it does not go far enough
in representing our complete understanding of the situation. So, consider the
following story:
Jack had finished painting. Father asked him if he washed the brush. Jack
said " N o , but Mother called me for lunch, so I left the brush in the paint
and will wash the brush later."
To understand why Jack had an acceptable answer requires knowing that the
crucial problem with paint on the brush is that the paint will dry, and this in turn
leads to the problem of stiff and unabsorbant brushes. Further, we must know
that drying requires exposure to air, and if the brush is in the paint it is not
exposed to air. This would seem to imply that we should include in our reason all
sorts of information about what happens when paint dries. But clearly we need
this knowledge in other situations also (paint dries on other things besides paint
brushes) so we should presumably put it into its own separate frame, P A I N T DRY. Space economy argues that this information should not be replicated
elsewhere, so we must somehow utilize P A I N T - D R Y in our explication of the

PAINT-DRY
EVENT : I---PAINT-DRY2(AND PAINT-DRY3 (paint on object)
|
PAINT-DRY4 (liquid gone leaving residue)
|
COMES-FROM: (evaporation))
[WITH-OPTIONAL-CONDITIONS
[(PAINT-DRY6 (object is absorbant)
|CAUSES
t.-PAINT-DRY7
(AND PAINT-DRY8 (residue part of object)
(IF PAINT-DRY6
THEN(AND PAINT-DRY10 (object is no longer absorbant)
PAINT-DRYI 1 (object is no longer flexible) ) ) )
FIG. 6 PAINT-DRY.

366

EUGENE CHARNIAK

reasons for washing. To see how this could be done, let us take a look at this new
frame.
This again is not complete, but for our purposes we only need to note that
PAINTING37 calls for a situation which is indirectly opposed to that called for
in PAINT-DRY3, assuming of course we identify the painting instrument in
PAINTING37 with the object in PAINT-DRY3. The reason for P A I N T I N G 3 7
then is to prevent P A I N T - D R Y from occurring on the instrument and hence
preventing the undesirable results which P A I N T - D R Y can cause. To put this
slightly differently, we do P A I N T I N G 3 7 because it L E A D S - T O the negation of
PAINT-DRY3 and hence the nonoccurrence of P A I N T - D R Y . This in turn
suggests casting our frame statement as follows:
PAINTING37

(NOT

(paint on painting instrument)

LEADS-TO: ( P A I N T - D R Y 3 ) )
COMES-FROM: ( W A S H - G O A L )

As before, the C O M E S - F R O M explains how P A I N T I N G 3 7 is to be achieved.

The LEADS-TO, on the other hand, is "attached" to the inner statement (paint
on painting instrument) and says that if one does leave the paint on, one might
have problems with the paint drying, provided the other prerequisites to
PAINT-DRY are satisfied.
A "reason" then is a description of the events that will occur, or fail to occur,
if the suggestion is carded out. We access this description by a pointer to the
frame describing this event, and in particular to the frame statement which the
suggestion tells us either to achieve, or to make sure never takes place. This
pointer is marked by the symbol L E A D S - T O rather than, say, " r e a s o n " for
much the same reason that we used C O M E S - F R O M rather than " u s e . " There
are cases in which "reason" has unacceptable anthropomorphic connotations, as
for example in PAINTING37 itself. It would hardly do to say that the "reason"
(or " g o a l " ) for leaving the paint on is to allow it to dry. (Note that the use of
G O A L in the representation (see Fig. 5) is teleological in nature, although much
of the difference between L E A D S - T O and G O A L is syntactic in nature.)
I have introduced L E A D S - T O as a way of indicating "reasons" because that
was my initial motivation, and it seemed the mo~t intuitively straightforward way
to explain them. There is, however, a possible misunderstanding stemming from
this about which the reader should be warned. Since "reasons" as I have been
using the term are causal relations, there is a temptation to interpret A LEADSTO B as A CAUSES B. This temptation should be avoided. A L E A D S - T O
pointer does not itself indicate causation, but rather points to where relevant
causal information can be found. A parallel statement can be made about
C O M E S - F R O M , but its introduction via P L A N N E R " u s e " statements does
not seem to invite the parallel misunderstanding.
To give another example of a L E A D S - T O tag, the frame which describes a
drip states that one of the prerequisites for dripping is the presence of sufficient

A FRAMED PAINTING

367

fluid sticking to a surface. So DRIP3 in the next example states that the volume
of this fluid must be greater than some threshold. It is this prerequisite which we
negate in P A I N T I N G 1 9 (not too much paint on instrument) and we will indicate
this by:
PAINTINGI9

(NOT

( G R E A T E R amount of paint on instrument

DRIP-CONST)
LEADS-TO: (DRIP3))

I have started in P A I N T I N G I 9 to put the body of the statement into formalism

so as to better indicate the correspondence between what P A I N T I N G 1 9 requires
and what DRIP requires.
We noted at the end of Section 1.2 that the C O M E S - F R O M tag could be used
to relate an activity to its underlying physical processes. The same is true of
L E A D S - T O . Within P A I N T I N G there are several examples of this, one of
which we have not yet included in our painting frame:
PAINTING26

( G R E A T E R amount of paint on instrument

COAT-CONST)

The reason we insist on P A I N T I N G 2 6 is that otherwise the paint will not cover
the portion of the object with which the instrument comes in contact. To put this
slightly differently, P A I N T I N G 2 6 ensures that STICK8 (see the sticking rule at
the end of Section !.2) is satisfied. We can indicate this by adding to P A I N T ING26
LEADS-TO: (STICK8)
(That this leads to the paint being on the object, as opposed to the instrument or
elsewhere is a problem which will be dealt with in Section 2.3.) Another example of the use of L E A D S - T O is P A I N T I N G 2 2 (bring the instrument in contact
with the object). This guarantees that STICK6 is satisfied, and hence should be
similarly marked. These changes have been made in Fig. 7 ( P A I N T I N G try 3).

1.4

Intermediaries and Matching

The skeptical reader, that is, the one who checks every example to make sure
they are as the author claims, may have noticed that some of the recent examples
using L E A D S - T O pointers simply do not work, at least not in the way explained
in the paper. To see this, the rest of us must be a bit more skeptical ourselves'and
look.at precisely what is being done. We have introduced the following into
PAINTING:
PAINTING22

(bring instrument in contact with object)

L E A D S - T O : (STICK6)

Looking back to the sticking frame we find:

STICK6

(Liquid in contact with object)

368

EUGENE CHARNIAK

PAINTING try 3
GOAL : PAINTING-GOAL (object to be painted has a coat of paint on it)
EVENT: -PAINTINGI4 (NOT (object dirty)
LEADS-TO: (DIRTY-OBJ3)
-PAINTINGI5 (everything is covered with paper)
-"PAINTING 16 (LOOP
-PAINTINGI7 (get paint on the painting instrument)
COMES-FROM: (STICKI0)
"PAINT1NGI9 (NOT (GREATER amount of paint on instr
DR1P-CONST)
LEADS TO: (DRIP3))
COMES-FROM: (REM-LIQ-GOAL)
-'PAINTING22 (bring instrument in contact with object)
LEADS-TO: (STICK6)
(DURING PAINTING26 PAINTING22)
-PAINTING26 (amount of paint on instrument is greater
than some constant)
LEADS-TO: (STICK8)

)
--PAINTING37

FIG. 7

(NOT (paint on paint instrument)

LEADS-TO: (PAINT-DRY2))
COMES-FROM: (WASH-GOAL)

PAINTING try 3.

The problem here is the requirement that if we have

FRAME-STATEMENTI

(body of frame statement 1)

LEADS-TO: (FRAME-STATEMENT2)

then the bodies of frame statements one and two must match. Now given my
propensity to express the bodies informally with an English phrase, it is hard to
decide precisely when two such bodies do match, but it should be clear enough in
this case that P A I N T I N G 2 2 and S T I C K 6 do n o t - - a painting instrument is a
solid not a liquid.
Of course, it is clear what is happening here. By the time we reach P A I N T ING22 we have paint on our instrument, so it is the paint which comes in contact
with the object, and it is the paint which should be matched against the liquid in
STICK6. So P A I N T I N G 2 2 and S T I C K 6 can be made to match if we somehow
use the auxiliary rule found in Fig. 8 when making the match. (The reader may
have noted that I used an arrow here to indicate an implication, whereas earlier
they were used to indicate time sequences. In fact, the arrows really indicate

STICKY-ON6 (AND STICKY-ON7 (STICK-ON LIQ SUR)

IMPLIES
STICKY-ON9 (CONTACT SUR OTHER-SUR) )

ST1CKY-ONI0 (CONTACT LIQ OTHER-SUR)

FIG. 8

A relation in the STICKY-ON frame.

A FRAMED PAINTING

369

nothing as they are entirely superfluous and not seen by the machine at all. I have
included them simply as means to provide some visual bracketing.)
I have used the full formalism in Fig. 8 because, as we have seen, informal
English makes it difficult to decide precisely when two statements match. The
essence of this rule should be clear enough, however. If a liquid is sticking to a
surface, and a second surface comes in contact with the first, then it also comes
in contact with the liquid.
Intuitively this rule is part of what we mean when we say a liquid is STICKYON a surface. There are naturally other such rules; for example, if something is
sticking to a surface, then the surface supports it. Or again, to stick is to stick on
the exterior of the object, and in a sense, to become part of that exterior. And no
doubt there are many more. These rules are gathered together in the STICKYON frame. Needless to say, S T I C K Y - O N is a state, not an action like painting
and all of the others we have looked at so far, and naturally state frames will have
somewhat different properties from action frames. One such difference is the
presence of rules like the one presented above. In Section 2 of this paper, we will
discuss state frames in detail, but for the moment we simply need a way to
indicate that this rule serves as an intermediary between P A I N T I N G 2 2 and
STICK6. We will do this as follows:
PAINTING22

(CONTACT AB-SUR-PART SURFACE-PART)

L E A D S - T O : ( S T I C K Y - O N 9 S T I C K Y - O N 1 0 STICK6)

( A B - S U R - P A R T is part of the absorbant surface of the painting instrument,

while S U R F A C E - P A R T is part of the surface of the thing being painted.)
In this representation ( S T I C K Y - O N 9 STICKY-ON10) are to be interpreted as
follows: to transform our starting point ( P A I N T I N G 2 2 ) into our end point
(STICK6), first match the starting point with S T I C K Y - O N 9 then utilize the rule
which connects S T I C K Y - O N 9 to S T I C K Y - O N I 0 , and match S T I C K Y - O N 1 0
against the end point (STICK6).
One could, of course, not include intermediaries in the representation and
instead expect the program to locate the appropriate rule when making the match.
I have not taken this approach because it seems to me that it is worth the extra
space in order to save the extra time needed to search for the appropriate intermediaries between our starting and finishing points. This will become even
clearer when we need two or three intermediary pairs to make the transition.
Indeed, let us consider such a case now. The example is the rule we introduced
saying that the amount of paint on the painting instrument should not exceed a
certain threshold for fear of dripping. As last seen in Fig. 7 it looked like:
PAINTING19

(NOT

( G R E A T E R amount of paint on instrument

DRIP-CONST)
LEADS-TO: (DRIP3))
COMES-FROM: (REM-LIQ-GOAL)

370

EUGENE CHARNIAK

We have not specified the precise form of REM-LIQ-GOAL, but the general
idea of REM-LIQ is to describe how one uses pressure on an absorbant object
(like a dish cloth) to decrease the liquid in it. As a first approximation then, the
goal of REM-LIQ would be:
REM-LIQ-GOAL

( C H A N G E LIQ-VOL amount ' N E G A T I V E )

which is intended to mean that the liquid volume changes negatively by some
amount.
Clearly PAINTING19 does not in fact match REM-LIQ-GOAL, and to
make them match we need three separate relations. One of these is seen in Fig. 9.
This rule says that if a thing increases (decreases) it may increase (decrease)
sufficiently to become greater (less) than a given amount. In the situation at hand
this means that we can use REM-LIQ to decrease the liquid volume sufficiently
to prevent dripping. But note--and this is one of the benefits of requiring explicit
matching--because the change rule only works sometimes (the value could
change, but insufficiently to raise it above a given threshold) it automatically tells
us that the utilization of REM-LIQ does not guarantee fulfillment of our goal. It
is somewhat similar to the situation with bringing the instrument into contact
with the object--as the rule of STICKY-ON explains, there must be paint on the
instrument to make everything work as desired. (Although in both cases the
envisioned program to use these rules would assume they went through unless
told otherwise.) So we can change PAINTING19 to read:
COMES-FROM: (CHANGE5 CHANGE2 REM-LIQ-GOAL)
But this is still not quite right, as we can see by comparing the bodies of
PAINTINGI9 and CHANGE5.
PAINTINGI9

(NOT

CHANGE5

( G R E A T E R amount of paint on instrument)

CONSTANT) )
(GREATER
X
THING)

This, however, can be corrected by a second intermediary shown in Fig. 10.

This rule which helps define G R E A T E R can be used in our intermediary as
follows:
COMES-FROM: (GREATER5 G R E A T E R I CHANGE5 C H A N G E 2
REM-LIQ-GOAL)

CHANGE2
IMPLIES
CHANGE3

FIG. 9

(CHANGE THING AMOUNT SIGN)

(SOMETIMES)
(IF (EQUAL SIGN 'POSITIVE)
THEN CHANGE4 (GREATER THING X)
ELSE CHANGE5 (GREATER X THING) )

A relation in the CHANGE frame.

A FRAMED PAINTING
GREATER1
IFF
GREATER2

E
FIG. 10

371

(GREATER A B)
(AND GREATER3 (NOT GREATER4 (GREATER B A) )
GREATER 5 (NOT GRE'ATER6 (EQUAL A B) ) )

A GREATER relation.

And, in fact, even this is not complete, but the general idea of how intermediary
rules link up in chains should by this point be sufficiently clear to make yet
another example unnecessary. (The missing rule would relate the amount of paint
absorbed by the instrument to the amount on the surface, which is actually
responsible for dripping.)
Let us step back a second and take a fresh look at what we are doing with our
intermediaries. Suppose we have:
FRAME-STATEMENTI
(...)
LEADS-TO: ( I N T I . . . INTn F R A M E - S T A T E M E N T 2 )
or, for our current discussion, the equivalent
FRAME-STATEMENT2 (...)
C O M E S - F R O M : (INTn . . . I N T I F R A M E - S T A T E M E N T 1 )
In both cases we are making the following claim:
Given the situation described in frame-statement 1, and given certain other
conditions, as indirectly specified by I N T l . . . I N T n , and all of these hold at the
same time T then (usually) the situation described in frame-statement2 also holds
at time T.
Another way to think of the relation between frame-statements 1 and 2 is to
consider them two different viewpoints (in the sense used by Bobrow and Winograd (1977)) for the same situation. So, to take one of the earlier examples,
bringing a brush into contact with an object can be seen from another viewpoint
as bringing the paint on the brush into contact with the object. Looked at in this
light, the intermediaries are translators between viewpoints. (This is, however,
only an analogy since Bobrow and Winograd's use of viewpoints differ in many
ways from my use of L E A D S - T O and C O M E S - F R O M . ) Naturally, viewpoints
can be more or less similar, and this would be accounted for by intermediaries
wtrich make more or less radical translations. For example, some intermediaries,
like the G R E A T E R relation we looked at earlier, have the. status of logical
relations, and in such a case the two frame statements are evidently just different
words for essentially the same thing. But in other cases, as with our rule of
S T I C K Y - O N , we seem to be at the border line between a rule of logic and the
physical laws of our world. In still other cases we have clear physical or chemical
laws relating our two frame statements. For example, one such law is PRESS

372

EUGENE CHARNIAK

which explains how pressing a liquid-saturated solid will cause an increase in the
amount of liquid on the surface of the solid. (It goes practically without saying
that when I talk of a "physical law" here I mean a "common sense physical
law.") The rule of PRESS is used in R E M - L I Q where it relates increasing
pressure on the solid to the change of volume of absorbed liquid. That is, we
would have:
R E M - L I Q - G O A L ( C H A N G E L I Q - V O L amount ' N E G A T I V E )
C O M E S - F R O M : ( . . . PRESS5 P R E S S 4 . . . REM-LIQ6)
where REM-LIQ6 is the order to up the pressure on the solid. The intermediates
in this case, including PRESS, relate two different viewpoints. On one hand we
have an increase in force, on the other we have a decrease in volume. It is the
purpose of the intermediaries to explain how such disparate things are really the
same.
2.

HOW TO PUT THE FRAMES TOGETHER

The first section of this paper introduced, in a somewhat impressionistic

fashion, many of the basic concepts which underlie the representation. In this,
the second section, we will essentially start at the beginning again, and perform
each step in a more methodical manner. The discussion will still fall short of a
formal description, however, and for those who feel the lack, a context free
grammar is provided in the appendix. To make it easier to see the correspondence between the rules there and the discussion here, I will occasionally include
references (e.g.,/l/) to the corresponding rule number. A program which incorporates these rules has been written and used to check all of the frames to ensure
that they follow the indicated formalism.

2.1

Frames and Frame Types

Since everything the representation can express is to be found in one frame or

another, the division into types of frames corresponds to the view of the world
assumed by the representation. At the highest level there is the usual division
between actions and states, and each of these comes in two varieties, giving us
four basic kinds of frames: complex event, simple event, state, and object.
(There is a fifth kind, the adjunct frame, which we will describe later.)/2/As we
shall see, the frame type determines to a large extent what makes up the frame.

Simple Event. The distinction between complex and simple events is the
distinction between P A I N T I N G and the common sense causal laws on which it
depends. Each simple event frame describes a single cause and effect relation,
whereas complex event frames are held together not by causal relations, but
rather by time relations, i.e., a typical complex event has the form, first you do
this, and then you do that, and . . . .
Complex events never contain causal
relations because there is no need for such relations. Presumably if certain

A FRAMED PAINTING

37,3

actions in a complex event lead to a certain result (by a causal relation) this will
be as the result of some common sense physical law which will be expressed as a
simple event. Rather than explicitly include the causal relation in the complex
event frame (which as we have seen will in most cases lead to unnecessary
duplication of information) we can instead include a L E A D S - T O pointer in the
complex event pointing to the simple event.
Now the above description tendS to make it sound as if the representation has
built into it a physicist's view of a deterministic world which runs according to
fixed causal laws. In fact, this is not the case. For one thing, the laws expressed
in the simple events may only sometimes apply (in which case there will be a
" S O M E T I M E S " marker by the cause statement/4/), and for another, there are
no particular restrictions on the kind of causal relations which may be expressed.
So, for example, a child's understanding of the workings of an electric light
might be expressed as a " c a u s a l " law of the form " t h e switch in the on(off)
position causes the light bulb to give (not give) light." Or for that matter one
could include a " l a w " which claims that kissing a frog sometimes causes it to
change into a prince.
We have already seen one example of a simple event, namely, S T I C K in Fig.
4, and we have another below in Fig. 11. (In Fig. ! 1, and henceforth, a " ; "
indicates that the rest of the line is comment, and not part of the frame.) Besides
those features which all frames share (i.e., a name, frame type indicator, and
variable list/t/) our simple events also have a section headed E V E N T which
describes what happens. As mentioned above, a simple event always consists of
a single causal relation in which the before and after sections consist of a single
formula/3/. (Note my use here of " f o r m u l a " as contrasted with "statement."
As I will use the terms here, a statement consists of a single predicate-argument
structure, plus name, plus tags. A " f o r m u l a " may simply be a statement, but it
may also be several statements connected together by the likes of A N D , OR,
etc.)

DIRTY-OBJECT (SIMPLE-EVENT)
VARS:
; This section specifies the variables.
; The actual format will be given in
; Section 3.3.
EVENT: [--DIRTY-OBJECTI
|
(AND DIRTY-OBJECT2 (PAINT-COAT PAINT OBJ SURFACE)
|
; a PAINT-COAT on the SURFACE of OBJ
|
DIRTY-OBJECT3 (DIRTY SURFACE) )
/CAUSES (SOMETIMES)
/(BEFORE (BEGIN DIRTY-OBJECTI) (BEGIN DIRTY-OBJECT4)
|
'(1 YEAR) )
t'-DIRTY-OBJ4 (OR DIRTY-OBJECT5 (CRACK PAINT))
DIRTY-OBJECT6 (FLAKE PAINT) )
FIG. I I The simple event DIRTY-OBJECT.
/

374

EUGENE CHARNIAK

Normally it is assumed that simple events happen virtually instantaneously as

is the case for STICK. But this is not true for D I R T Y - O B J E C T , which is
intended to express the fact that if one has a coat of paint over a surface which is
dirty, the paint is likely to crack or flake. That the cracking or flaking takes about
a year to occur is expressed in the event by the time relation. ( B E F O R E (BEGIN
D I R T Y - O B J E C T I ) (BEGIN DIRTY-OBJECT4) ' (1 YEAR)). (The one year,
as with all numbers used in the frames, hides an implicit "although this can vary
by a factor of four.")
Other allowable time relations include D U R I N G and S A M E - T I M E , and
naturally enough one can talk about the ending time of states and actions as well
as the beginning/5//6//7/.

Complex Events. As for complex events, they describe an event in terms of a

collection of formulas with time relations interrelating the formulas/10/. Complex events also have a section which simple events do not, namely, G O A L ,
which describes the normally desired resulting state of the event [8[. Many
complex events, however, do not have a goal in this sense, a typical one being
tennis, which is usually done for its own sake. In such cases the goal will be
marked instead with the special frame O W N - R E W A R D . For goal-directed activities the G O A L will always have a C O M E S - F R O M pointer into the event.
This serves to describe how the event produces the desired goal/9/. For example,
in PAINTING we will have to include a pointer from the goal state to PAINTING22 (bring instrument in co~tact with object). The intermediaries on this
C O M E S - F R O M pointer will in effect state how it is that bringing the brush into
contact with the object serves to coat the object with paint.
States. A state frame describes a state much as an event frame describes an
event. Naturally there is no activity involved, but as we noted in Section 1.4,
there are various relations between the given state and others. So, for example,
we pointed out in Section 1.4 how STICKY-ON related to other states like
S U P P O R T (A sticky-on B ~ B support A) and E X T E R I O R (A sticky- on B
A is on the exterior of B). At the moment all relations are of the form A
IMPLIES B, or A IFF B (IF and only IF), although it may have a modification
such a s " s o m e t i m e s " attached to indicate that the rule may fail to hold /12//13/.
These relations are gathered together under the heading R E L A T I O N S and
constitute the bulk of most state frames (see Fig. 12). There are, however,
besides R E L A T I O N S , two optional portions to state frames, C O M E S - F R O M
and L E A D S - T O / 1 1 / . These have much the same purpose here as they do in
their more familiar role as additions to frame statements. For example, the
C O M E S - F R O M tag in the S T I C K Y - O N frame points to the normal method for
achieving STICKY-ON, namely, the rule of STICK (seen in Section 1.2).
LEADS-TO gives common results, or uses, of the state. To take another case,
ABSORBED-BY (Fig. 13) L E A D S - T O points to the rule of pressure causing
more liquid to come to the surface.

A FRAMED PAINTING

375

ABSORBED-BY (STATE)
RELATIONS: vABSORBED-BYI (ABSORBED-BY FLU SOL)
LI MPLIES
ABSORBED-BY2 (SUPPORT SOL FLU)
ABSORBED-BY3 (ABSORBED-BY FLU SOL)
MPLIES
BSORBED-BY4 (STICKY-ON SURFLU SOL-SUR)
COMES-FROM: ABSORPTION5 ; this describes how a liquid which
; is sticking to an absorbant object
; will come to be absorbed by it
LEADS-TO: PRESS2
VOL-CHANGE2

; describes the previously mentioned

; rule of pressure
;
;
;
;

describes how adding or subtracting from total absorbed fluid

will also change the volume on
surface

FIG. 12 The ABSORBED-BY frame.

Objects. As for object frames, such as those describing paintbrushes, rollers,

etc., I will have little to say about them. They are most similar to state frames,
but the relations portion is replaced by a D E S C R I P T I O N portion which, naturally enough, gives a physical description of the object/14/.
Adjuncts. An adjunct frame, as the name is intended to suggest, does not
give a complete frame, but rather additions to, and modifications of, another
frame (henceforth called the " m a s t e r " frame). Perhaps the best examples here
are the frames which describe how to use a brush, or roller, in painting.
P A I N T I N G as I have noted, is neutral as to the precise instrument used. But
neutrality is not sufficient, for we do have knowledge which is quite specific to,
say, using a roller for painting, and this knowledge must be expressed somewhere. For reasons I will discuss at the end of this section, I have chosen to put
this knowledge into a frame of its own. If this frame is to describe how a roller is
used in painting, it will need to refer to our general knowledge of painting, and
since it will obviously not do to repeat these facts over again, the new frame must
make heavy use of the old one. To get some idea of the necessary interactions,
consider some of our specific knowledge of roller painting.
a. The instrument used is a roller.
b. Before starting to paint, transfer some paint into the roller tray.
c. Get paint on roller by rolling it in the tray.
d. To remove excess paint, we still utilize R E M - L I Q , but we also apply
pressure to the various portions of the roller surface by rolling the roller along the
inclined section of the roller tray.
e. The contact of the roller with the painted object is done while rolling the
roller along the surface.
What we have here is a series of comments on the original painting frame. By

376

EUGENE CHARNIAK

and large they have no particular internal coherence, but rather each is to be
understood only in relation to a specific portion of the.master frame, which is
specified by the subsection of the frame labeled M A S T E R / 1 5 / . (See Fig. 13,
below, for an illustration.) To reflect this dependence, the E V E N T portion of,an
adjunct frame will consist of "modification statements" each of which have two
sections, the first identifying the portion of the master frame that is to be elaborated, the second giving the elaboration. So, fact (c) above would go as follows:
PAINTING22

C O M E S - F R O M : (ROLL3)

Here the left hand, or identification, portion of the modification statement

specifies that it is formula P A I N T I N G 2 2 in P A I N T I N G which is to be modified, and in particular the C O M E S - F R O M slot within PAINTING22. (For the
purposes of adjunct identification one can think of formulas in a frame as LISP
atoms, each having a name (like PAINTING22) and a property list with properties, the most obvious being C O M E S - F R O M and L E A D S - T O / 1 8 / . And, in
fact, the frame checker (Appendix) translates them into exactly this formalism.)
PAINTING22 in the above case is the name of the statement in P A I N T I N G
which describes the instrument in contact with the object, while ROLL3 in the
right-hand, or elaboration, portion of the adjunct statement, is the portion of
ROLL which describes how part of the rolling object comes in contact with the
path over which it is rolling. The total import of the statement then is that the
contact of instrument with object comes about, should the instrument be a roller,
in the process of rolling the roller. Note that the modifications given are considered additions to anything which already appears in the indicated slot of the
master frame, not replacements. (For P A I N T I N G 2 2 C O M E S - F R O M , above,
this is irrelevant because the slot was not filled in the master frame.) However,
should a complete replacement be desired, this is indicated by the symbol REPLACE/17/. (See P A I N T I N G I 7 in Fig. 13.)
ROLLER-PAINTING(ADJUNCT) try 1
MASTER: (PAINTING where the instrument variableis always bound to a roller)
EVENT : (DURINGPAINTINGI6 ROLLER-PAINTING3)
: ROLLER-PAINTING3(FLUID,CONTAINMENT PAINT TRAY)
REPLACE PAINTINGI7 COMES-FROM
: (ROLL3)(where rolling is done along bottom of tray)
REPLACE PAINTING19 COMES-FROM
: (REM-LIQ-GOAL) (where REM-LIQ is done on the inclined portion
of the tray)
PAINTING22 COMES-FROM: (ROLL3)
FIG. 13 ROLLER-PAINTINGtry I.
Besides indicating that a slot of the master should be filled, an adjunct frame may
also add new formulas to the time sequence given by the master/16/. For
example:

A FRAMED PAINTING

377

(DURING P A I N T I N G I 6 R O L L E R PAINTING3):
R O L L E R - P A I N T I N G 3 ( F L U I D - C O N T A I N M E N T PAINT TRAY)
In such cases the identification portion gives a time relation to previously established formulas, while the modification portion gives the new formula/I 9[, Most
of these points are illustrated in Fig. 13. The portions of the frame expressed in
English (where . . . ) require aspects of the representation which we will not
encounter until later.
Let me conclude here by contrasting my adjunct frames with two somewhat
different approaches to roughly the same problem. The first of these is seen in
(Schank et al., 1975) where diversity of specific cases within an overall scheme
are handled by different "tracks" within the same overall "script" (equivalent to
my " f r a m e " ) . My reasons for not adopting this approach are well illustrated by
taking the Schank example, eating in a restaurant. In a short time I was able to
come up with 20 different tracks through the restaurant frame, each differing at
some point in the manner or time sequences of seating, ordering, obtaining food,
and paying. (I would not be surprised if spending a day at it produced yet another
twenty.) Admittedly some of these tracks only applied to a single restaurant, but
I see no reason why this should exclude them. My objection to including all of
this in a single frame then is that the frame will have to include too much
information irrelevant to the particular case, or particular restaurant, at hand.
This will, of course, make searching the frame more time consuming.
An approach more similar to mine is that taken by Bobrow and Winograd
(1977) with their "prototypes." Indeed, their approach seems more elegant than
mine, but I doubt that it is sufficiently flexible, since a prototype, as far as I can
tell, can only be modified by giving it different fillers for its "slots." (Roughly
speaking their "slots" correspond to variables which are given as arguments to
the prototype.) As my roller painting exampl~ illustrates, unless everything is a
slot (variable), then this approach cannot work. And even if we made every
statement in the frame a variable, it is by no means obvious how we could then
handle situations like R O L L E R - P A I N T I N G 3 (see Fig. 14) where we wish to
add on an extra action to be performed. (But having never used the language I
may be overlooking something.)

2.2

Loops and Continuity

In the last section we distinguished between statements and formulas, where

the latter included the former, but also included combinations of statements
connected in some fashion. In this section we will specify more precisely the
structure of formulas.
A formula may simply be a frame statement, but if not, it consists of an
optional name, and a formula body/20[. In general, we will only bother to name
a formula if for some reason it is referred to elsewhere. (It should be noted that
naming in itself is of no theoretical interest, since in a completely debugged

378

EUGENE CHARNIAK

ROLLER-PAINTING(A.DJUNCT)
MASTER: (PAINTING (ABSORB-PART ROLLER-COVER) )
EVENT : (DURING PAINTINGI6 ROLLER-PAINTING3)
: ROLLER-PAINTING3 (FLUID-CONTAINMENT PAINT TRAY)
LEADS-TO: (FLUID-CONTAINMENT6 FLUID-CONTAINMENT9) STICK6 STICK10 STICKY-ON7)
REPLACE PAINTING17 COMES-FROM
: (STICK10STICK6 STICKY-ONI0 STICKY-ON9 PARTIAL
(ROLL3 (ROLL (CIRCUMFERENCE ABSORB-SUR) (PATH
TRAY-BOT) ) ) )
REPLACE PAINTING19 COMES-FROM
: (variousintermediaries
(REM-LIQ-GOAL(REM-LIQ (LOCATION TRAY-PLANE) ) ) )
PAINTING22 COMES-FROM: (ROLL3)
FIG. 14 The ROLLER-PAINTING frame.
system we could uniformly replace names by pointers to the formulas, or statements, themselves. This has not been done because the resulting circular list
structure would be difficult to debug and awkward to put down on paper.) The
formula body is simply one or more subformulas connected together by AND,
OR, IF T H E N ELSE (where the ELSE would be optional), NOT, L O O P , or
C O N T I N U O U S /21/ /22/. Except for the last two, these are all fairly
straightforward, and will not be discussed further.
We have already seen a L O O P statement in P A I N T I N G , and as its name
implies, it is used to indicate a situation where the same activity is repeated one
or more times, with some condition changing each time. The only things which
in fact change in LOOPs are the values of certain variables, called "loop variables," which are expected to change value each time through the loop. They
will be discussed further in the next section. In the example we have seen in
P A I N T I N G the loop is for replenishing the paint on the painting instrument, and
the loop variable, i.e., that which changes, is the particular blob of paint which is
sticking to the painting instrument.
The reader may have already noted one strange feature of LOOPs, and that is
that by the usual sorts of conventions the loops here are nonterminating, in that
they have no stated conditions for termination. This, however, is more apparent
than real, and in general a loop may be terminated in one of two ways. The first,
and most obvious, is that the G O A L of the complex activity (all LOOPs are in
complex activities) has been satisfied, and the loop was instrumental in bringing
this about. Again, P A I N T I N G serves as a good example. The goal of PAINTING is for the object to acquire a coat of paint. As we noted in Section 2.1, this
goal will have a C O M E S - F R O M pointer to P A I N T I N G 2 2 along with intermediaries explaining how bringing the instrument in contact with the object will
achieve part of this goal. Since PAINTING22 is within the loop P A I N T I N G I 6 ,
this loop will be terminated upon completion of the goal. Note that a particular
goal may be abandoned before it is fulfilled, and this would have the same loop
terminating effect.

A FRAMED PAINTING

379

Second, often loops, or statements within a loop, will have associated rules of
the form "while you are doing that loop, make sure this side condition holds." In
such cases, should the side condition fail to hold, then the loop is discontinued
until (and unless) the side condition can be reestablished. So, for example, if
Jack is painting something outdoors and the wind blows away the newspaper
from underneath the object (i.e., PAINTING15 ceases to hold), we might expect Jack to suspend painting until the paper can be repositioned. If this proved to
be sufficiently difficult, as if the paper had completely blown away and Jack had
no more, we would no longer consider this merely an interruption, but would
instead interpret the goings-on as a complete halt to the painting activity, and
hence to the relevant loops.
Similar to L O O P is " C O N T I N U O U S " which indicates that certain steps are
looping " c o n t i n u o u s l y " in the sense that there are no longer discrete
iterations--e.g., the " s t e p " in E V A P O R A T I O N which says that the volume of
liquid decreases. Or again, in painting, the contact of the instrument with the
object is continuously changing. That is to say, what we previously represented
as

PAINTING22

( C O N T A C T INST-SUR OBJ-SUR)

should actually look more like:

PAINTING21

(CONTINUOUS
PAINTING22 ( C O N T A C T INST-SUR OBJ-SUR)
PAINTING23 (DISTANCE OLD OBJ-SUR
CONTINUOUS)

Here OBJ-SUR is the " l o o p " variable and PAINTING22 says that we are
interested in where the instrument touches the object. PAINTING23 tells us in
what sense OBJ-SUR is changing continuously. There m a y also be further
constraints on how the continuous loop variables change, but they are not needed
in PAINTING21 /23//24//25/.
Continuous actions may appear in both complex events and simple events. In
the former we break a continuous action just as we break a loop. For example,
the continuous action represented by PAINTING21 will be broken for exactly
the same reason that the loop P A I N T I N G I 6 would be--the G O A L is completed or abandoned, or a side condition causes a possible permanent interruption
in the action. However, a continuous action in a simple event will not have goal
states to interrupt it in this fashion, and will instead be interrupted by other
simple actions, such as a falling object coming in contact with some surface.

2.3

Frame Statements and Their Tags

While we have thus far considered frame statements in some detail, there are
still a few aspects which we have not introduced. For the most part these deal
with the tag, and in particular with special cases of the use of C O M E S - F R O M

380

EUGENE CHARNIAK

and LEADS-TO pointers. However, some of the additions are more basic, so let
us start with them.
First, a minor point--on some occasions we will allow the name of a flame
statement to stand in for the entire statement/26/. This is particularly useful in
the next section where one statement will serve both as the definition of a
particular variable while at the same time playing a role in the event or relations
which are the heart of the frame.

Variable Bindings. A frame statement in effect says, " w e have here an

instance of a certain frame (say, STICKY-ON) which has in this case the
following values for its variables." In all of the examples so far, the variables
have been specified in positional notation. For example,
PAINTING17

(STICKY-ON S O M E - P A I N T ABSORB-SUR)

In these cases it is assumed that the order given in the list of arguments (e.g.,
(SOME-PAINT ABSORB-SUR)) corresponds to the ordering of the variable
list found in the frame (to be described in Section 2.4) just as in most programming languages.
However, a second method of indicating argument bindings is allowed, and
this is to indicate them by pairing the variable name with its value, as in:
(STICKY-ON

(SUR ABSORB-SUR)

(LIQ SOME-PAINT) )

Note that when the bindings are given by pairs the ordering is of no relevance.
One may also start the argument list using positional notation, and finish using
pairs, but not vice-versa. Lastly, no fixed number of arguments need be specified
in order to instantiate a frame/27//28//29//30/.
It has proved convenient to allow both of these notations. On one hand, most
frames, and this applies especially to state frames, have certain arguments which
are most crucial, and hence are almost always specified. By allowing these to be
specified in place notation we do without the excess space required by pair
notation. On the other hand, many frames (and here I am mostly talking about
event frames) have a substantial number of variables, of which only a few will
ever be specified for any given instance of the ~ctivity. That is, if we arereading
about painting, the story might talk about the part of the wall covered with paint
on a particular stroke, in which case the variable representing this would be
bound to the appropriate constant in the story representation, but this would be,
in general, quite unlikely. Since there are many variables of this sort, to assign a
place to each such variable, and require that it can only be bound by an expression which also binds each prior place, would be quite unwieldy.
Intermediary Variable Binding. This leaves the tags, that is, the things like
our COMES-FROM and LEADS-TO
pointers. In our discussion of them so
t
far, one problem we have ignored is how the variables in the pointed to frame get
bound. Consider, for example:

A FRAMED PAINTING

PAINTINGI9

(NOT

381

( G R E A T E R S O M E - P T - V O L DRIP-CONST)
LEADS-TO: ( D R I P 3 ) )

This tells us that if there is too much paint on the instrument it can lead to
dripping, and in particular it would satisfy the rule of dripping which gives as a
prerequisite that there be sufficient liquid on the surface of an object. But if we
wish to use the information in DRIPPING in order to make inferences then we
must somehow bind the variables in DRIPPING to their appropriate values,
which will be derived from PAINTING.
The reason this problem has been ignored up till now is that I have been tacitly
assuming the typical pattern matching variable binding convention. That is, in
the process of matching the inner portion of PAINTING19 to the corresponding
statement in DRIPPING we will match, and bind, the variables in DRIP to the
corresponding ones in PAINTINGI9. So, if the corresponding statement in
DRIP is:
DRIP3

( G R E A T E R LIQ-VOL DRIP-CONST)

in making the match we would bind L I Q - V O L to S O M E - P T - V O L , and

DRIP-CONST (in PAINTING) to DRIP-CONST (in DRIP), since this is the
only way the match can be made. This type of binding works in the vast majority
of cases, but not always. In some cases there will be crucial variables in the
subsidiary frame (i.e., the one pointed to by the L E A D S - T O or COMESFROM pointer) which can indeed be specified by the calling frame, but which
do not appear in the frame statement, and hence will not be bound in the course
of the match. One example of this is found in our R O L L E R - P A I N T I N G frame,
a complete version of which is found in Fig. 14. When we first looked at this
frame in Fig. 13 we had the following rule:
R E P L A C E PAINTING 17 COMES-FROM
(ROLL3) (where rolling is done along bottom of tray)
This is intended to express how one gets paint on a roller. If we actually look at
the two relevant frame statements we will see:
PAINTING17 (STICKY-ON S O M E - P A I N T ABSORB-SUR)
ROLL3 ( C O N T A C T C I R C U M - P T PATH-PT)
A quick look shows us that we need intermediaries to bridge the gap between the
two, and they will effectively say that we can bring about STICKY-ON by
establishing C O N T A C T with a liquid, which in turn is done by establishing
C O N T A C T with a surface which has liquid on it. This is all well and good, but
in making this transfer via the intermediaries we are not told which path is being
used, since the variable which gives this information (i.e., which says that the
path is the bottom of the tray) is not included in PAINTINGI7. To include this
we will 6hange the R O L L E R - P A I N T I N G statement to the following:

382

EUGENE CHARNIAK
R E P L A C E PAINTING17 C O M E S - F R O M
: ( (ROLL3 ( R O L L ( C I R C U M F E R . E N C E ABSORB-SUR)
(PATH TRAY-BOT) ) ) )

(Again the intermediaries have been left out for the sake of simplicity.)
What we have done then is to add on an extra frame statement that will serve to
bind the variables in R O L L should we want to make inferences based on it. If we
now add the appropriate intermediaries we get:
(STICKI0 STICK6 STICKY-ON10 STICKY-ON9 P A R T I A L
(ROLL3 ( R O L L ( C I R C U M F E R E N C E ABSORB-SUR)
(PATH TRAY-BOT) ) ) )
The parentheses which enclose ROLL3 and its frame statement simply serve to
"attach" the frame statement as a modification of ROLL3 /34/. ( P A R T I A L
here is a symbol which states that the subsidiary frame statement does not
completely satisfy the calling frame statement. In this case, the statement in
R O L L says that the rolling object has one point in contact with the path. Obviously this is insufficient for paint to be acquired all over the roller surface, but it
is a start. This same confusion exists when we relate getting paint on one part of
an object to coating the object.)
Other Tag Indicators. So far we have only been considering the case where
a frame statement C O M E S - F R O M or L E A D S - T O a single other frame statement. In the case in which there is more than one possibility we connect up the
different possibilities with AND (when they all occur) and OR (when normally
only one occurs)/32//35/.
Lastly, besides C O M E S - F R O M and L E A D S - T O we have two other tags,
UP-DATES and U P D A T E D - B Y , which, as their names suggest, indicate how
one frame statement may cause another no longer to be true/31[. They have the
same form as C O M E S - F R O M and LEADS-TO, except that since they always
point to a single statement, the A N D and OR forms are not used/36//37/.

2.4 Variables and Variable Types

Variable Specification. One aspect of the representation we have so far
steadfastly ignored is the nature of the variablds we have been using so freely. By
and large we have skirted the problem through the use of mnemonic names for
our variables so it would be reasonably clear exactly what sort of things they
were. So, for example, in our R O L L E R - P A I N T I N G adjunct frame, we had a
variable named R O L L E R which naturally is intended to be bound only to things
which are rollers. But variable names in the system are in fact arbitrary, so at
some point we must state this restriction on the values of R O L L E R .
This becomes even more evident, and more problematic, when the restrictions
are not so tight. Consider, for example, the variable I N S T R U M E N T which is
the instrument used in painting. It is tempting simply to say that it must be either
a roller or a paint brush, and while this will work for the overwhelming majority

A FRAMED PAINTING

383

of cases, it will not work for all, since one could easily use, say, a sponge or a
wash cloth. Indeed, given the choice between a sponge in good condition, and a
paint brush which was permanently caked with so much paint as to be brittle,
none of us would have any hesitation in taking the former. So perhaps we should
say that I N S T R U M E N T should be bound only to things which are both flexible
and absorbant. But then, what about a story like:
Jack had to paint a chair. Finding he had no brush he proceeded to dip his
hand in the paint and then smear the paint along the chair.
One's hand is flexible enough, but not very absorbant, and indeed, it is not too
hard to imagine a story in which someone painted a wall using a spatula, which is
not even flexible. When one comes down to it, aside from the fact that the
instrument must be a solid, there are very few absolute constraints on what it
must be.
But neither can we simply say that the instrument must be a solid and leave it
at that. Notice the difference in the following two stories.
Jack had to paint a chair. He dipped
into the paint.

~a rock
}
t a paint brush

In the second case, but not the first, we immediately understand what is going
on. That is to say, we may allow rocks to serve as painting instruments, but
before we recognize them as such we need quite a bit of circumstantial evidence,
and in particular much more evidence than we need to see paint brushes in the
same role.
At the very least we seem to need two levels of restrictions. The first are those
which absolutely have to hold, while the second are those which, while usual, are
not strictly required. (This corresponds pretty much to the distinction between
absolute and preference restrictions found in (Hayes, 1976).) The former will
usually just rule out obvious impossibilities, while the second gives positive
evidence for matching a given variable against a certain object. (It seems likely,
however, that this simple bifurcation of constraints will ultimately have to be
made more complex. My reasons for thinking so stem primarily from the observation that deciding when an object in a story should be bound to a frame variable
is simply a special case of the recognition, or diagnosis, problem. That is to say,
deciding that a particular object is serving as a painting instrument on a given
occasion has much in common with deciding that certain pieces of evidence
imply"the existence of a given disease, or that certain visual clues imply that one
is looking at a certain object. The recognition problem is obviously a very tough
one, however, and we will say no more about it here.)
One portion of our frame (labeled VARS), then, will be devoted to defining
our frame variables and the values they may assume. This portion will simply be
a listing of variable entries/38/in which each entry will in turn consist of the
mandatory variable name (everything else is optional), followed by a type decla-

384

EUGENE CHARNIAK

ration, the absolute r.estrictions, and finally the normal restrictions, which will be
indicated by the symbol N O R M A L / 3 9 / . Of all of this, the only thing left to
discuss is the type declaration, which tells the system under what circumstances
the variable in question is allowed to change values.

Multiple Valued Variables. One of the most important features of variables

is that in a certain sense they do not vary at all, or at least not all that much. That
is to say, given two appearances of the same variable in the same program, they
will have the same value, unless the program explicitly changes it. Indeed, in
normal mathematics, two appearances of the same variable in the same equation
will always have the same value. And an ideal variable in our frames would
behave in exactly the same way.
To see why this would be ideal, consider the process of deciding if a particular
story statement matches a particular frame statement. Of course we can immediately check to see if the predicates match, but this is hardly sufficient.
CONTACT, for example, appears at least five or six times in PAINTING and
surrounding frames. We must also check to see if the variables in the frame are
consistent with the arguments given in the story. This process is simplified if our
frame variables are bound, since we then need only check if their values are
identical to the arguments given in the story. But note that this will only work if
variables never change their values. If they can, then failure of the story statement to have exactly the right arguments might only indicate that one of the
variables in the frame has changed its value.
But however advantageous unchanging variables might be, the system must
allow them to change value. In particular, it will recognize three different ways
in which such value changes might occur: explicitly marked change variables,
for-all variables, and normal change variables/40/.
Explicitly Marked Change Variables. Of these we have so far only encountered the explicitly marked change variables. When we discussed loops and
continuous loops in Section 2.2, we noted that they would have loop variables
associated with them. A loop variable is simply a variable whose value normally
changes after each iteration of the loop. That-a given variable is in fact a loop
variable is simply indicated by marking, in the type portion of the variable entry,
the name of the loop for which it is a variable. These variables are "explicitly
marked" in the sense that there is one precise place in the frame where a variable
change is allowed, and this place is marked in the variable entry. A second type
of explicit change is a C H A N G E statement within a frame. For example, we
have already noted the need for a simple event frame which will express how a
liquid absorbed in a material will come to the surface when pressure is applied to
the material. This will be expressed as seen in Fig. 15. The important lines, from
our current point of view, are lines PRESS4 and PRESS5 which express the
change in pressure and the change in surface volume, respectively. The variables

A FRAMED PAINTING

385

PRESS (SIMPLE-EVENT)
VARS: LIQUIDI
SOLIDI

(LIQUID LIQUIDI)
PRESS~ (ABSORBANT SOLIDI)
LEADS-TO: (ABSORPTION2 ABSORPTION4 PRESS2)
SUR-SOLIDI
(SURFACE SUR-SOLID1 SOLIDI)
PRESSUREI
TYPE: (PRESS4) ; This variable changes its value at PRESS4
PRESS3 BELOW ; i.e. PRESS3 given below defines
PRESSUREI
S1GNI
PRESS4 BELOW
SUR-LIQ-V~)L TYPE: (PRESS5)
(VOLUME SUR-LIQ SUR-LIQ-VOL)
SUR-LIQ (AND (STICKY-ON SUR-LIQ SUR-SOLID1)
(PART-OF SUR-LIQ LIQUIDI) )
PRESS2 (ABSORBED-BY LIQUIDI SOLIDI) ; This states that
EVENT: PRESSI (AND
PRESS3 (CONTACT SUR-SOLIDI (PRESSURE
PRESSUREI) ) ; increasing
PRESS4 (CHANGE PRESSUREI (SIGN SIGNI) ) )
; (decreasing) the
CAUSE
; pressure on a soaked solid causes the amount
PRESS5 (CHANGE SUR-LIQ-VOL (SIGN SIGN1) ) ; of surface fluid to increase
; (decrease).

I
L

FIG. 15 The PRESS frame.

PRESSUREI and VOL-SUR-LIQ are the ones which change, and this is indicated in their entries by the type: (PRESS4) (for VOL-SUR-LIQ we have
PRESS5 instead) which says that the change which occurs in the value of the
variable occurs due to PRESS4 (or 5).
For-All Variables. The second of our three types of allowable value change
is our for-all variables. As their name indicates they are used in situations in
which one would normally think of the predicate calculus for-all quantifier. So,
for example, I distinguish between a S U R F A C E , which means what one would
expect, and an EXTERIOR, which one does not, as the paper we spread on the
floor is an E X T E R I O R of the floor, but it is not the S U R F A C E of the floor.
(All SURFACEs are EXTERIORs, however.) Some of the relations which
define E X T E R I O R include:

( E X T E R I O R E X T OBJECT1)
IMPLIES (SOMETIMES)
( B E T W E E N EXT S U M T H I N OBJECT1)

This says that for the most part the exterior will be between anything else,
SUMTHIN, and the object itself. S U M T H I N here is a for-all variable which
means, in this case, that it may have any number of values. In the variable entry
we will indicate that it is a for-all variable with the symbol F O R - A L L .
In other situations, and in particular when the for-all variable appears in the
antecedent portion of the rule (or of a simple event), the for-all variable must take

386

EUGENE CHARNIAK

on all acceptable vallaes before the rule can be considered satisfied. For example,
in the simple event F A L L we will have the condition
FALL2

(NOT

( S U P P O R T S U P P O R T E R OBJECT) )

Here S U P P O R T E R is a for-all variable, and the net effect of this rule would be
roughly equivalent to a M I C R O - P L A N N E R " T H N O T " statement. That is,
the rule in effect says, " m a k e sure you cannot find something which supports the
object." Note that if S U P P O R T E R were simply a normal variable the rule
would simply say "see if you can find something which does not support the
object.' '
Normal variables. Finally, we have our so-called " n o r m a l " variables,
which are unmarked by an explicit type declaration. The properties of normal
variables within complex events (and adjuncts to complex events) are quite
complex, so let us at first ignore that case. For all other types of frames normal
variables are quite simple--once they have been bound for a particular instantiation of a frame they may not change their value. That is to say, once in PRESS
we have bound, say, S O L I D I , it may not receive another value within the
particular instantiation of the PRESS frame. If something else is pressed, then a
new instantiation will be made. (This ignores the problem of mistakes. Should
the system mistakenly bind a normal variable it may, of course, correct itself.)
With complex events, however, normal variables may, in certain situations,
change their values. The need for this is not too hard to see. Suppose that Jack
starts to paint a wall with a roller. Initially, then, the normal variable INSTRUM E N T will be bound to the roller. What do we do then should Jack at some
point switch to a paint brush? We might possibly refuse to change the value of
I N S T R U M E N T and instead create a new instantiation of the P A I N T I N G
frame, this time with I N S T R U M E N T bound to the paint brush, but this certainly goes against one's normal analysis of the situation. Jack, after all, has not
done two lots of painting, one with a roller, and one with a paintbrush. Rather,
Jack has done one lot of painting using both. Furthermore, there are good
efficiency reasons for going along with our intuition. After all, most of the
variables in the two frames will have the same bindings; the object will remain
the same, as will the paint and the paper used to cover the adjacent objects. And a
little thought will show that the other normal variables (the paint used, the object
painted, the paper used to cover the adjacent objects, etc.) also may change their
value. The change of one need not imply the change of the others, and in general
does not so imply. Hence they, too, must be allowed to change within the same
instance of PAINTING.

Dependency Change. There is one last way in which variables may change
their values. Often a variable is defined (i.e., restricted to certain values) in terms
of other variables. A typical example is a variable that is to represent the volume
of a certain liquid which will be defined as follows:

A FRAMED PAINTING

387

SUR-LIQ-VOL ( V O L U M E SUR-LIQ SUR-LIQ-VOL)

Naturally in such cases, should SUR-LIQ be allowed to 'change values, then
obviously SUR-LIQ-VOL must change values also. Such cases will not be
marked explicitly, but will rather be indicated by the presence of a second
variable in the variable entry. Note that any type of changing variable may
receive new values in this way, provided only that the variable it is dependent on
may itself change. (Hence this would not apply to variables dependent on normal
variables in states or simple events.)
3.

CONCLUSION

There are several traditional roles which a conclusion can play in a paper such
as this. One is to indicate the nature of the future research suggested by the
foregoing analysis, and indeed, I should mention that I am writing (Chamiak,
1977) a program that will utilize the frame representation described here. In
particular, I expect that this program in its early incarnations would accept simple
stories about painting (such as the one given in Section 2. I) which have been
hand parsed into a semantic representation. The program would answer questions
such as the ones found in that example, again with the output in semantic
representation, not English. Even when this program is written, of course, the
result would be in many respects considerably cruder than programs which
already exist. But as I explained in the Introduction, I would hope that a program
utilizing the frames described here would have a deeper and more complete
understanding of the subject it was discussing then any other program to date.
Naturally, there is much other work which needs to be done, from making
improvements in the existing formulations of PAINTING and its subsidiaries to
extending the coverage to other topics (and no doubt also extending the representation), or worring about how more complex stories (such as typical children's
stories) might be handled by such a system. But it seems to me that before such
work can be carried out I must have a better feel for the limitations of my
approach. The PAINTING complex has now become sufficiently elaborate that
its computational implications are no longer very clear to me. Hence the need for
a program.
With that statement, let me now take advantage of yet another traditional role
for a conclusion, and discuss a topic which, while separated from the main trends
of the paper and hence unmentioned until now, is at the same time sufficiently
bound up with what I have been saying to warrant a brief discussion. The topic is
semantic representation.
Until now, except for a tangential mention in the first paragraph of this
section, the words "semantic representation" have not been used. The reason is
that strictly speaking our concern here is not the representation of what we are
told in language but rather the representation of what we already know. Yet given
the generally recognized need to link the former to the latter, there can hardly be

388

EUGENE CHARNIAK

a strict separation of.these topics, and if the words "semantic representation"

have not come up, it reflects more on the necessities of formal presentation than
the underlying structure of what was being presented'.
As is well known, there are two trends in the semantic representations used by
researchers in artificial intelligence. One of these, represented by Schank (1975)
and Wilks (1976) sees semantic representation in terms of a small number of
so-called "primitives" which can be combined to form meanings for the expressions in natural language. The other trend, illustrated by Bobrow and Winograd
(1977) and Moore and Newal (1973) sees semantic representation rather in terms
of a variety of predicates, at many levels of generality, and perhaps even overlapping each other in order to provide for different aspects of the same natural
language expression. Note, by the way, that I have called these "trends" rather
than "positions" for by and large there are no well defined positions taken by
either camp. Indeed, given some formulations of these trends there is really
nothing to argue about. Sometimes the primitive trend is portrayed as simply
claiming that on some occasions, some English words are represented, in part,
by predicates which are more general than the full meaning of the word. (E.g.,
"drive" might include a semantic predicate like " m o v e " in its representation.)
It would be virtually impossible to disagree with such a position, and many
positions within the nonprimitive trend are similarly unobjectionable (and the
position I am about to take is sufficiently hedged to make it equally bland). But
nevertheless, the two trends are clearly separable, and I would imagine that even
given the lack of prior discussion, the reader may easily figure out where I would
fit into this division. But rather than simply telling, let me try to show where I
stand in all of this.
If we were told that Jack was painting a chair with a certain brush, a system
using the representation given here would need to recognize that an instance of
PAINTING was in progress. Ignoring the problem of correctly locating the
event in time (begun, but not yet finished) this would involve setting up an
instance of the painting frame with the proper variable bindings in order that we
can see i f later statements correspond to the actions one would normally expect
from a painting event. So we might require something like:
PAINTING1

(PAINTING JACK1 CHAIR1 BRUSH1)

Here JACK1, etc., represent the specific things mentioned in the story. Now
what precisely is the status of PAINTINGI? In the preceding lines I have gone
slightly out of my way to present it as technical maneuver, the purpose of which is
to ensure that the right variables in PAINTING are bound to the right objects in
the story. And indeed, PAINTINGI does serve this purpose. But is there any
reason for not considering PAINTING1 the "semantic representation" of the
original English sentence? (Again barfing the question of time representation.)
One usual argument against things like PAINTING1 being taken seriously as
a semantic representation is that it explains nothing--we have simply replaced

A FRAMED PAINTING

389

the English word "painting" by a "semantic concept" P A I N T I N G - - a n d except for giving the objects different typefaces, the entire exercise is circular.
However, this argument does not hold against PAINTINGI because the symbol
PAINTING has a lot more to back it up than its fortuitous resemblance to an
English word. There is the framed PAINTING. Furthermore, given the representation I have outlined so far, there is every reason to see P A I N T I N G I as at
least part of a semantic representation for our sentence. After all, practically
everything in the system has the form of PAINTING1. That is to say, if you
wish to express something in this system, you have to apply a frame to some
arguments, and the way you do this is by a construction of the same form as
PAINTINGI. If other examples of this form are to be considered semantic
representation, why not PAINTINGI ? (There is the opposite tactic of including
PAINTING as one of our primitives. But to do so would mean that the number
of primitives is quite large, to a first approximation about the size of a typical
person's passive vocabulary. I take it as axiomatic that allowing this many
"primitives" would simply make the entire debate go away.)
But this is not to say that PAINTING1 is necessarily the complete semantic
representation of our sentence, for there might well be compelling arguments for
always including more information which, although derivable from P A I N T I N G ,
would be worthwhile having already available without making the inference. So,
to switch examples, suppose we are told that Jack drove to town. A standard
primitive analysis would have as its center something like J a c k went to town by
using an automobile. To this would typically be added further information about
how or why, but we already have enough to see one advantage over the representation being advocated here, that is, a call to the driving frame.
DRIVING1

(DRIVING JACK1 T O W N I )

The advantage is the way the primitive representation highlights the notion of
motion, and hence the idea of Jack's being in town (or the likelihood thereof).
Facts like this are often important in the further analysis of the story, so we may
well be creating extra work by burying the information inside DRIVING. This
suggests instead a general rule that (AT A G E N T L O C A T I O N ) (in the appropriate modality) should always be included in our representation of driving events,
even though strictly speaking it is redundant. Perhaps we might even see fit to
generalize this to all goal states of complex events.
.Perhaps there might even be other cases in which more "primitive" elements
are automatically included in the representation of certain concepts. But the
tendency illustrated here is clear. It is things like PAINTING1 and DRIVING1
which are primary, for it is they which give the maximum amount of information
about the goings-on in the story. Our general or primitive concepts are to be
included sparingly, where their presence is clearly beneficial in the ever present
tradeoff between the storage of redundant information and the time it takes to
make an inference.

390

EUGENE CHARN1AK

Hence, to sum up both these last few points and by default the entire paper, I
believe that the apparent complexity of what we say in natural language is not a
disguise for the basic simplicity of a small number of concepts arranged in a
multitude of ways, but rather reflects the real complexity of the world we live in
and our knowledge of it. We understand by relating what we are told to what we
know, and given that the latter seems to be represented by concepts at all levels
of generality, the former should be also.

ACKNOWLEDGMENTS
I would like to thank the Dalle Molle Foundation, and the Depa~ment of Computer Science,
Universityof Geneva, who supported this research, the departmentand its director, Prof. B. Levrat,
for the use of their machine in order to programthe frame checker mentioned in the Appendix, and
Maggie Kingand Walter Bischof, for their detailed commentson the first draft of this paper. Finally,
I would like to reverse the parallel acknowledgmentfound in (Wettler, 1977).

APPENDIX
This appendix contains the formal grammar for the formalism described informally in the second part of the paper. A program has been written to check
that all frames conform to the grammar given here, and to translate the frames
into what I hope will be the computer usable form.
In keeping with the notation used earlier in the paper, anything in capitals
.corresponds to symbols which actually appear in the frames, i.e., the terminal
symbols, while nonterminals are in lower case. As for the rest of the notation,
"---~", " [ " , and " * " , have their usual meanings, but parentheses do not, given
that they are terminal symbols. Instead, optional elements are indicated by enclosing " / " . The non-context-free portions of the grammar are indicated by
English expressions describing what they do, and are enclosed'by " " .
Finally, the language given here differs in three minor ways from that given in
the body of the paper. In each case the change is due to the fact that in LISP
based systems it is more convenient to use parentheses to disambiguate expressions than formatting, such as I used previously. Hence the frame checker does
not require formatting, but does require three extra sets of parentheses, one
around each frame, one around each variable entry, and one around each relation. Otherwise everything is the same.
1.

2.

frame ~ (name (frame-type) variable-list core)

the nonterminal " n a m e " is not defined below, but is simply a Lisp
atom.
frame-type --~ C O M P L E X - E V E N T I S I M P L E - E V E N T [ S T A T E I
OBJECT I A D J U N C T

391

A FRAMED PAINTING

3.

core simple-events--~ EVENT: formula/WITH-OPTIONALCONDITIONS (statement*)/CAUS ES/

modifier/time-relation formula
all optional condition statements must be named, and each name
must appear in the condition portion of an IF formula in the final
formula of the event (i.e., the result of the event).

4.

modifier--~ (SOMETIMES)

5.

time-relation --~ (BEFORE time-arg time-arg/(number time-name)/

(SAME-TIME time-arg time-arg)
(DURING time-arg name) '1 ( D U R I N G name name)
number is of course supposed to be a number, while the names are all
names of formulas in the frame.

6.

time-arg---* (BEGIN name) I(END name)

7.

time-name---~SECOND
I YEAR

8.

core complex-~vents~

9.

goal --* (OWN-REWARD) I statement

the statement must be part of the complex-event-event, or else must
C O M E - F R O M a statement in the event.

10.

complex-event-event--~ formula I time-relation [ formula complex-eventevent [ time-relation complex-event-event

11.

core states --~ RELATIONS: relation*/COMES-FROM: comesfrom/ /LEADS-TO "a leads-to/

12.

relation ~ (formula relation-connector/modifier/formula)

13.

relation-connector--~ IMPLIES

14.

core objects-~ DESCRIPTION: description-form/COMES-FROM:

comes-from/ /LEADS-TO: leads-to/

15.

core aaJunct--~ MASTER: statement EVENT: adjunct-statement*

16.

adjunct-statement--~ slot-form I time-form

[MINUTE

[HOUR

[DAY

[WEEK

[MONTH

GOAL: goal EVENT: complex-event-event

[ IFF

17.' slot-form --~ /REPLACE/ name slot-name: slot-filler

"name" is the name of a formula in the master frame; "slot-filler" is
not defined below because it is completely indescribable in context
free terms. See Section 2.1 for a rough description of its permissable
forms.
18. slot-name~ A R G U M E N T S ] F O R M U L A - A R G U M E N T S I COMESF R O M I LEADS-TO

392

EUGENE CHARNIAK

19.

time-form ~ tirlae-relation: statement

the time relation must relate the statement on .the right hand side of the
adjunct statement to either a formula in the master frame, or a statement in the adjunct which is itself related to a formula in the master
frame.

20.

formula ~ / n a m e / f o r m u l a - b o d y

21.

formula-body ~ (AND formula-arguments) I ( O R formula-arguments) ]

(IF formula T H E N formula/ELSE formula/) I ( N O T
formula) [ ( L O O P complex-event-event) ] ( C O N T I N U OUS continuous body)
note that a N O T followed by a statement is a statement, not a
formula.

22.

formula-arguments ~ formula*

23.

continuous-body ~ statement continuity-st* constraint*

at least one of the variables in the statment must be marked as a continuously changing variable, and there must be a continuity statement
for each such variable.

24.

continuity-st ~ statement
but the special symbols OLD and C O N T I N U O U S must appear as an
argument in it.

25.

constraint ~ statement
but the special symbol O L D may appear as an argument.

26.

statement --~ name I/name/statement-body/tags/ ] name B E L O W

a name appearing by itself must be defined (complete with body) earlier,
unless BELOW comes next, in which case it will be defined below.

27.

statement-body ~ (name arguments)

28.

arguments ~ arg*

29.

arg ~ value (name value)

statement
the name must be that of a variable in the frame whose name is the predicate of the statement. Also, once the second form has been used, the
first cannot be used within the same statement.

30.

value ~ name 's-expression

"s-expression" of course is simply any LISP s-expression. The name
must be that of a variable which is "effectively local" to the frame where
the value appears. To be effectively local to a frame X, a name must
either be defined in X, or if X is an adjunct frame, the name must be
effectively local to the master frame.

] statement

](NOTstatement-body)

I$ST

A FRAMED PAINTING

393

31.

tags--~ C O M E S - F R O M : comes-from I L E A D S - T O : leads-to

U P D A T E S : updates I U P D A T E D - B Y : updated-by

32.

comes-from ~

[(OR

33.

basic-comes-from ~ (intermediary*)
PUNISHMENT) [
(OWN-REWARD)
there is a check on the predicates of the original statement (i.e. the one
whose tages we are now examining) as well as those of all the intermediary statements to ensure that matching is at least in principle possible. O W N - P U N I S H M E N T and O W N - R E W A R D do not occur in
C O M E S - F R O M , see below.

34.

intermediary --~ P A R T I A L [name/name/(name/name/statement)

the one name version can only be the last intermediary. The two names
must be frame statements in the same fram. e, and if there is a statement,
it must have this frame as predicate.

35.

leads-to--~ comes-from
but keep track of the fact that we are dealing with a leads-to, since the
predicate check mentioned above depends slightly on this fact.

36.

updates ~ basic-comes-from
for the sake of the predicate check record that we are dealing with an
update.

37.

updated-by --~ basic-comes-from

record that we are dealing with an updated-by.

basic-comes-from [ ( A N D basic-comes-from*)
basic-comes-from*)

[(OWN

38. -variable-list ~

V A R S : variable-entry*

39.

variable-entry ~

(name/TYPE: type/

/formula/

40.

type ~ name [ F O R - A L L
" n a m e " must be that of either a C H A N G E
formula, or a C O N T I N U O U S formula.

/ N O R M A L : formula/)
statement, a L O O P

REFERENCES
Bobrow, D. G., & Winograd, T. An overview of KRL, A knowledge representation language.
Cognitive Science, 1977, 1. 1-46.
Charniak, E. A partial taxonomy of knowledge about actions. In Advance papers of the fourth
international conference on artificial intelligence. Tbilisi, 1975. Pp. 91-98.
Charniak, E. Inference and knowledge, part II. In E. Charniak & Y. Wilks (Eds.), Computational
semantics. Amsterdam: North-Holland, 1976. Pp. 129-155.
Charniak, E. Ms. Malaprop, a language comprehension program. In Advance papers of the fifth
international conference on artificial intelligence. Cambridge, Mass., 1977.

394

EUGENE CHARNIAK

Hayes, P. J. A process to implement word-sense disambiguation. Working paper 23, Institute for
Semantic and Cognitivo studies, 1976,
Moore, J., & Newell, A. How can MERLIN understand? In L. W. "Gregg (Ed.), Knowledge and
cognition. Hillsdale, N.J.: Lawrence Erlbaum Associates, 1973.
Rieger, C. The common sense algorithm as a basis for computer models of human memory,
inference, belief and contextual language comprehension. In R. Schank & B. L. Nash-Webber
(Eds.), Theoretical issues in natural language processing. 1975. Pp. 180-195.
Schank, R. C. (Ed.). Conceptual information processing. Amsterdam: North-Holland, 1975.
Schank, R. C., & the Yale AI Project. S A M - - A story understander. Yale University computer
science research report 43, 1975.
Wettler, M. How to connect frames. Working paper 29, Institute for Semantic and Cognitive
Studies, 1977.
Wilks, Y. Parsing English, part 11. In E. Charniak & Y. Wilks (Eds.), Computational semantics.
Amsterdam: North-Holland, 1976. Pp. 155-184.