Name: David Lynch

Title: The structure and mathematical form of visual language representation

Targeted audience:
Editorial for someone curious about visual informatics
Natural Language: A disapointment
When I was a small child I remember asking my mom how to spell my name. I
was really trying to find out what the structure of language was. She wrote it on the
counter. I had to climb onto the counter to see it. A few minutes later, after pressing my
mom with questions, I realized that the human language that my world was centered
around sucked. I thought to myself, “How could anyone convey what they mean through
reading or writing words?” I thought that there should be much grander ways of
communicating. It seemed way too slow to read one word at a time. It seemed like
reading might be a good thing for a snail to do, but not a human. I was honestly very
disappointed. I was only slightly comforted by the fact that reading would be so simple
to learn since it had the same format as spoken words. We all have come to accept
natural language as the status quo for communicating. However, before I looked on the
counter, I did not. I have resented natural language ever since! It is still my long
outstanding goal to fix language by designing a much richer, more powerful, and efficient
language.
What can we expect from language?
From dictionary.com the word “language” is defined as “a system of signs,
symbols, gestures, or rules used in communicating.” It is also defined as “a particular
manner of expression.” Natural language does not effectively utilize those defining goals
of language. However, it fits a third definition very well: “The communication of
thoughts and feelings through a system of arbitrary signals.” This is just the case, as
natural language is more of a signaling device than it is a language, and an arbitrary
signaling device at that. It’s through human ingenuity that we figure out what each other
mean, not due to help from natural language. We must look beyond a language that
simply strings together thousands of different identifiers. A language should rather have
fewer words and more interactive language structures. The purpose of having a language
is to be able to express complicated things by using smaller parts. However with natural
language, that building process is left to the reader. Natural language does not directly
contribute to the building process of an idea. It only plays a passive role. As well, the
language must not play a passive role with regard to the unique information being
presented. A language should be explicitly intertwined with the information being
presented by that language. Natural language does not treat information with respect. It
does not care who or what it is describing. It has no desire to understand the information
it is describing. It sits passively in the background like an impersonal narrator with a
mouth that never stops talking. Natural language operates only as a signaling device, not
as a constructable bridge to understanding. I believe it is time to wash that mouth out
with soap, so I can quit listening to its ineffective squalls to be heard.
The problems with natural language are thus deeper than its surface. But its
surface is what makes its problems so obvious, since conveying an idea in words can be
such a long and indirect effort. Expressing information through language is like
following branches on a tree with little way to reference back to the root. Language
should instead aid our ability to navigate and flow through information. We often think of
language today as the artifact from stringing words together. Though, the real measure of
a language is in its ability to facilitate the flow of information, not impede. A most
obvious limitation of natural language is the inability to build new words on the fly. One
often finds themselves looking for through a thesaurus for the right word, only to come
up empty-handed. How pathetic! Does one find this in itself problematic? Yes.
On building a more dynamic language
Flowing through information in a more recursive and interactive manner would
mean deriving multiple meanings from one piece of information. It would mean
returning to a previous point in a language expression, after handling new information.
An entire expression must work together to form a system that is not uni-directional, as is
natural language. A dynamically flowing language would be similar to a computer
program or algorithm. The structure of a computer program is carved to fit its purpose.
Language must be carved around information in just the same way. This means that a
language based on nouns and thousands of identifiers must be replaced with a form that
uses patterns and metaphorical leverage. This will allow information and ideas to be
treated as small abstract structures that can be handled in a more constructional manner.
On using visual representations of language
Representing such algorithmic language systems could be aided by designing
visual information spaces. Since patterns would be such a big part of this new language
design, it would seem favorable to utilize very visual representations. Developing visual
language systems raises a whole host of new issues which include the language issues
first discussed. The main problem with visual information is that people still must read
an image a section at a time. It is not practical that one could take in the entire message
of the image at one time. Even with a photograph, a person must study different parts of
the scene separately. However, visual information offers boldly different and powerful
benefits over reading text. However the underlying structure of the language itself would
remain the main bottleneck. Language and information must be molded in a way that is
easiest to process once it becomes harnessed in one’s mind. Language need not be left on
the paper. It should be usable in one’s mind and thought. It is then, of second
importance, to structure and arrange information according to a flow that would best
accommodate the process of visually extracting it. Information can be structured in
largely different ways, while still meaning the same thing. This lends considerable
flexibility for developing visually effective representations. One would initially think of
visual diagrams as great for representing very repetitive patterns, such as the factorial of a
number. However, the factorial of a number hardly represents the type of information
that intimately depends on language. Repeating patterns are unnecessary though, since
the reader need not learn the same thing over and over. However, with the coupling of
repeating patterns and internal variations, one can more easily read a visual message.
While one can absorb some overall structural properties of a visual representation, they
must be able to read through it in pieces as well. A flow of information can be seen
below, where items are essentially ordered in a way that can be read from left to right.

One could group together similarities in many overlapping ways. For instance, all
elements with a “d” could be placed on the outer edge. Such rules could allow the reader
to quickly visually identify many overlapped and unique relationships. Such visual
languages may offer a powerful new way of navigating through information. The 3rd
dimension could be used for time. It is as well very likely that a reader could interact
with the visual representations in a dynamic way.
 The mathematical properties of shapes and patterns can play a powerful role in
the development of visual language systems. One might imagine that sophisticated
geometric structures could represent information in a very pure way. It would be relevant
to ask how certain sets of patterns could be mathematically packaged into visual
representations. One would be interested in succintly visually representing sets of
interdependent patterns. Since sets of random or related information may have larger or
indirect logical similarities or relationships, it becomes important to find global
mathematical parrallels that could be demonstrated through fluid visual ensembles.
While information could be packaged based on patterns or themes in the data, it is often
necessary to separate information into independent components. But, as well,
information is often intertwined into what one could consider a very dynamic algorithm.
An algorithmic sort of information model is interesting to think about in terms of
mathematically congruent visual models. While it is unlikely that a continuous
congruency between the shapes is going to aid a reader much all at one time, it is possible
that a reader could continue asking new questions about the model. Clearly a simple set
of combined shapes could have parallels among the shapes that would be enough to
confuse a reader eternally. That is why the reader is not asking questions about the
model, as much as they are reading the model. Meaning, the mathematical complexity
would be scaled back to accommodate the desired complexity of the language
representation itself and the desired readability of the visual representation. But on the
other hand, mathematically based spatial images would offer quite a curiosity, since it
could mean representing algorithms in context with some of their mathematical
boundaries. As well, redundant information could be used in the visual representation to
aid readability. The true property of language involves flowing between ideas. This
would require the dimension of time to be included. Reading visual information would
thus involve navigating through disjoint information sets over time.
2d visual representations are very demanding for the reader, who will constantly
struggle between identifying larger symmetries, and reading through very unique local
information sets. Planar symmetry groups demonstrate the capacity to use global
similarities and overlapping symmetry. Multiple reflections and rotations are shown
below.
 One could easily imagine modifying such visual information to combine
algorithmic information with these scaffolds. Rules could be described and followed
throughout regions of the visual structure. Below you can see a triangle formed from
three green boxes. The forming of a triangle might initiate a rule that asked how many
black squares lie within the triangle. Such logical rule based control by the visual
environment would offer a new ability to engineer complex events. Redesigning the
visual system to highlight the most relevant information would be a very dynamic and
ongoing process.
 A more diagrammatic visual representation as shown below offers strengths and
weakness’ compared to those visual represenations that use rigorous mathematical
parallels.

The diagrammatic form can take advantage of color, line lengths, angles, and globally
linked information in a less mathematically relevant way. Its strengths are simply that it
is not as mathematical. More relevance can be placed on grouping objects into areas.
While generic grouping seems powerful, it quickly becomes limiting when the quantity
of groups outweigh the other structures operating in the diagram. It is of course
impossible to precisely gauge the effectiveness of all these examples, without developing
the underlying language and the particular informational substance. However, when
global, local, color and grouping variations are not present in an equal ratio; it becomes
less relevant to utilize them all in the same diagram. This is a powerful rule of thumb for
gauging the effectiveness of a visual representation for readability. This readability value
could be calculated quantitatively simply by measuring informational complexity against
the visual business. For instance, there may be little value in utilizing a square in the
image below among a massive quantity of symbols that control the bulk of the
informational substance. However, be aware that the exact size and position of the
square add at least two degrees of informational complexity. If the language were
designed to be biased using certain structures, then the presence of a square may be very
rare, and thus contain a richer meaning. However the game of building many unique
identifiers would quickly become a hieroglyphic nightmare, leaving one begging for
textual representation.

Clearly new mathematical questions arise when designing language systems that
use adjacently packaged shapes or package shapes within shapes. Firstly one must avoid
clutter, and overcomplicated mathematical parallels. This does not mean that the
mathematical consistency through a representation cannot be rigorous or mathematically
complex. The readers’ awareness of the mathematical complexities should only be
afterthoughts to otherwise straightforward interpretations of visual shapes. Below are
simple demonstrations of visual packaging that does not leverage well for informational
extraction. The random arcs and lack of geometry make it hard to read, and identify
global properties, even though artistically these M. C. Escher drawing may be very
intriguing. Notice that it is hard to transition between the otherwise symmetrical shapes
in the snake drawing.
 Packaging shapes within shapes can be a mathematically strong way to produce
visual systems. The locally readable structures must be separate from the globally
readable structures. Otherwise the reader will be forced to stare at the structure, trying to
find a connection between local structures and global structures. The example below
could quickly make a reader dyslexic and forgetful of which orientation they had read. It
is therefore necessary to compliment such heavily packaged and randomly arranged
information with some other information that offers a source of patterns.

It actually becomes easier to parse this image when it is connected as shown below.

The connected image allows the reader to find global patterns, such as the repeating
bottom right corners. Such rearrangements can help organize the components for the
reader. A language centered on patterns would have strong mnemonic features, such that
that following one memory reference to another could be eased. There would be many
possible ways of remember the informational structure’s that would be invisible to
someone not aware of the mechanism’s and rules used by the language. The visual
language could use position and locality to help show similarities within the information
in a very intuitive way. One could imagine setting up a visual representation so that
someone could quickly scan it for a particular type of information. In the simple diagram
below, it is easier to see on the left than on the right, that there are two sets of four and a
one set of six. One could imagine many opportunities to organize information to take
advantage of ones ability to eye-ball parallel information sets that may be important as a
mnemonic but possibly extraneous to the actual content of the expression being
conveyed.

Using shapes and geometry can be an effective way to refer to and make apparent
overlapping patterns and other largely packaged informational complexities. Visual
language systems offer a dramatically different and powerful alternative way for
accessing language. Most visual representations today consist of objects linked by lines,
whether it is cities or atoms. Such representations are only very schematic ones and do
not represent new forms of language. Languages have not been previously developed to
handle such interesting spaces. Discussion herein address’s some rules of thumb for the
major obvious limitations of different approach’s toward visually packaging information.
The idea of packing more and more symbols into a visual representation is counter
productive. Various types of visual data must be incorporated in equal ratios in order to
be effective. Otherwise one type of visual information will dominate the readers’ visual
space more than another set of visual information. This provides a motivation for
utilizing global geometric parallels. This facilitates the readers’ capacity to migrate from
one area to another as they read the properties within a visual representation with many
components. Further developments within such a language model could include
navigating and interacting through the visual information dynamically. A reader could
ask a question simply by modifying the shapes. This model clearly offers possibilities for
interactivity with language that would be impossible with natural language text.
 References:

David E. Joyce Wallpaper Groups.

Department of Mathematics and Computer Science
Clark University
http://www.clarku.edu/~djoyce/wallpaper/wall9.html

Doignon, Jean-Paul. Falmagne, Jean-Claude. (1999) Knowledge Spaces. Springer-Verlag

Berlin Heidelberg.