IV. THEORY DEVELOPMENT II

PROCESSES OF INTERACTION

The previous chapter explored the dynamics of the self-organization of stuff in a singularly complex way, parallelling it with systems in, amongst others, ecology. But there is something to resolve, as stuff itself does not move. In the end, it is only possible to explain the process through the interaction with living and acting human beings. 

In this chapter a model is developed to respond to this problem, linking the configuration of stuff to the cognitive abilities of people. Before looking at global patterns, we zoom in on the most local decisions; the act of picking up a pen, of moving chair or setting the table. According to action identification theory, these little moments of decision are not always conscious processes, but simply become automatic subordinate components to our larger goals (Vallacher & Wegner, 1987), and thus can self-organize into action. 

4.1 Stuff – Action – Identification

4.1.1 Affordances as a medium of interaction

Logically, stuff is related to activity, as it makes activity possible. However, it is not the individual things that offer us much; when we think about it, it is striking how little we can do with a bowl, a corkscrew or a bag. These offer the human body the somewhat useless possibility to ‘hold them’ or ‘rotate them’, but their actual service seems first to be directed to other artifacts, such as soup, a bottle with a cork or groceries that need to be carried. It is in a configuration that the actual emergent properties arise; combined with a spoon we can eat the soup, with a chair, a table and glass of water we can have a meal, and as more bowls of soup and chairs are added we can enjoy a dinner with friends. The more particles self-organize into a stuff cell, the more the configuration affords us.

Affordances is a term used in perceptual psychology, first explained by J.J. Gibson in the article ‘The theory of affordances’ (1977) and explored more deeply in ‘The ecological approach to visual perception’ (1979). Affordances of the environment are, Gibson defines, “what it offers the animal, what it provides or furnishes, either for good or for ill” (1979). Whereas classical perceptual psychology considers the mind as a black box that receives information, processes it and subsequently performs an action, ecological perceptual psychology links perception to action in a direct and complementary relationship (Gaver, 1991). It is therefore that this term lately became commonplace in interaction design, and is being explored in the field of architecture.

What Gibson suggests is that these possibilities of action are also what is perceived. Through a constant analysis of the environment in terms of affordances, our perception is not sensory-based, but information-based; we do not perceive stairs, but the affordance of stair-climbing (1979). This approach, that as a result makes the perception of the environment dependent on personal physical abilities as well as cultural values (e.g. strength, skills, beliefs) proved to be of value in explaining complex, open-ended events in everyday life (Gaver, 1991). A complementary relationship between the acting organism and the acted-upon environment is inherent to the term.

Artifacts as a medium of interaction, as described by Portugali (2016), can therefore be understood in terms of affordances; they are both an action possibility and a perceived action possibility. On the concrete and physical level of individual artifacts this is quite direct; a handgrip affords to be held and communicates accordingly. This interaction is often consciously planned by the designer and awaits to be performed by its user. The configuration of multiple artifacts, however, is essentially dynamic. The displacement of stuff is a continuous ad-hoc assemblage of affordances by the inhabitant itself. Stuff forms a landscape of affordances, from concrete to abstract, and therefore resembles a city, where texts, buildings and roads are external representations of ideas, intentions, memories and thoughts (Portugali, 2016), yet in a very individual, personal and day-to-day manner. By creating action possibilities we create the perception of action possibilities and thus communicate with ourselves.

4.1.2 Action identification as a medium of interaction

All action and all action possibility is related to internal ideas, cognitive representations of what one is doing or could be doing: action identities. The direction of this relationship has been a theme that in classical psychology has split opinions (Vallacher & Wegner, 1987). By some the cognitive identification of the action is regarded as something reflective that is constructed through a judgement of the situation (a.o. self-perception theory and psychoanalysis), while others see it as a template for subsequent behavior; after a mental representation of what one is going to do is constructed, the action is performed (a.o. psychology around the control of muscle movements). Both domains seem not to be able to explain the phenomenon of self-organizing stuff; an either top-down or completely bottom-up approach both miss the two-way interdependence that is necessary.

Action identification theory, introduced by R. Vallacher, presumes that as soon as we step out of the laboratory, both our thoughts and the sequence of actions we perform are anything but calm and straightforward. The mind is a turbulent and chaotic theatre of thoughts where at a fast pace all kinds of ideas, complementary, contradictory or even completely unrelated, pass by. Zooming in on the actual process of doing things the relationship between everyday actions and our thoughts on them might be not unidirectional but cyclical (Vallacher & Wegner, 1987) (Vallacher & Kaufman, 1996). The theory of action identification explores the idea of a causal interdependence between the two, on the local level of small and simultaneous decisions, that could be the key to defining complex global patterns.

The underlying concept is that every activity can be described in multiple ways. One can, for example, be ‘preparing a salad’, but could also be ‘eating healthily’,  ‘chopping tomatoes’ or ‘being creative with left-overs’. These identities find themselves in a hierarchical linkage, on a scale from higher level to lower level descriptions, defined by the words ‘by’ (higher to lower) and ‘to’ (lower to higher); one prepares a salad by chopping tomatoes by moving a knife up and down by using the muscles in one’s hand. One makes coffee to get energized to have a productive day (Goldman, 1970). Higher and lower level actions have quite different characteristics. Whereas low level identities express (muscle) movements, high level identities describe the action in relation to its context, often suggesting a larger meaning, goal or significance (Vallacher & Wegner, 1987).

People in principle tend to maintain their activity with regard to the current prepotent action identity in mind (Vallacher & Wegner, 1987). The cognitive concept of what one is doing regulates the behavior as it distinguishes suggestions on the continuation of the activity as either relevant or not, which means that it serves as an order parameter structuring the action. Moreover, when a low level activity is prepotent, people are eager to adopt any clue for a higher identity when it pops up (Vallacher & Wegner, 1987), even when the high levels have questionable connections to the lower level details (Vallacher & Kaufman, 1996). This sensitivity to clues for a higher level identity (e.g. in the form of an idea on the relevance of their action in a larger context) makes the process path-dependent; a relatively random occurrence can entirely change the course of the system. The emergence from a mechanistic depiction of behavior into a single comprehensive identity, the theory suggests, is a product of volatile mental activity, a global pattern arising from relatively wild and uncontrolled local interactions.

“ Such instances of instability represent more than noise or breakdown in a system. To the contrary, far from being unavoidable at best or dysfunctional at worst, instability plays a critical role in the functioning of many different kinds of systems. Simply put, the fluctuations among different states and patterns characterizing the unstable system provides the raw material for subsequent self-organization in the system. ” 

– Vallacher & Kaufman (1996, p. 265)

4.1.3 Affordances as external memory

Action identities are a constant reminder of what we are doing and serve as an order parameter structuring the activity after being emerged from a miscellany of thoughts and ideas that together form more and more abstract concepts. In all these respects they are similar to affordances. Affordances having emerged from concrete to abstract in the same way, at the same time being a communicative medium that appears to us as something identified; a corner for reading, a desk for drawing, a window sill for plant nursing. Affordances and action identities work complementary in the self-organization of action, as internal (cognitive) and external (physical) phenomena of the same kind. This implies that the configuration of stuff is an ad-hoc constructed extension of the brain, like a life-sized drawing, to-do list or post-it note.

Fig 4.1: Artifact configurations as an artificial external representation of an idea (i.e. an action). It is constructed specifically, yet out of existing objects and thus has characteristics of both a sketch and artifact. Adaptation of Stolk (2015), and Portugali (2002).

This can be illustrated by imagining coming home after a day at work, some free time ahead and no particular plan in mind. What if some robotic system would every day completely tidy up the living room by putting all items in order, stacked on the shelves and hidden behind cupboard doors? The space would feel uninspiring and as if it could have been everyone’s. It is when you come home that you want to see things and be given ideas, opted by the external memory from before you left. “I can read the paper! There was a project I was working on! That CD I bought new, I should still listen to it!”

4.2 LOWER AND HIGHER LEVELS

The first main principle of action identification theory holds, as we have seen, that action is maintained with respect to its prepotent identity. The second states that there is a tendency to jump to a higher level identity if possible; when constant focus on one’s fingers while playing the piano becomes unnecessary, people are likely to describe their activity as ‘making music’ or ‘entertaining friends’. Lastly, the third principle holds that when action cannot be maintained in terms of its prepotent identity, there is a tendency for a lower level identity to become prepotent (again). This mostly occurs in one of two scenarios. Either something goes wrong, the piano player makes a mistake and is suddenly forced to focus on the exact placement of their fingers again (unintended), or a task is too difficult to understand as a whole, and is therefore split up in smaller tasks to perform step by step (intended) (Vallacher & Wegner, 1987).

Lowering levels is quite easily caused by stuff, and the concept is often explained so. When something breaks, for example, gets lost or is used up, it disturbs the ongoing activity and becomes the focus of attention. An experiment conducted by Wegner et al. (1984) shows how odd tools inhibit people from reaching higher levels at all; coffee drinkers given heavy and oddly shaped cups described their activity as ‘lifting the cup’, while others given normal cups talked about ‘taking a break’ and ‘getting energized’. In other words, one can say that an action identity cannot be reached or maintained when corresponding affordances, the external action possibilities, are not or no longer present.

Fig 4.2: The external action possiblities (affordances) and internal action identity correspond. When the action possibility ‘breaks’ (e.g. caused by the malfunctioning of stuff) the action identity can no longer be maintained. Generally, but not always, the attention moves to how the action possibility can be restored (e.g. by repairing, replacing or refilling something).

Higher levels are reached by thoughts and suggestions on the activity in a larger context, that can especially in the volatile first phase be quite random and even unrelated to the matter (Vallacher & Wegner, 1987). We can add to this notion that not only internal ideas (fig 4.3), but also external ideas in the form of affordances fulfill this role (fig 4.4). Stuff that lies around can unexpectedly generate ideas and be of value, and thus lift the activity to a higher level. Especially starting from low levels this process seems to indeed happen almost by chance. When entering your home after work, it can be either an article in the paper, a book lying on the coffee table or a radio program that happens to first grab your attention. When doodling on a piece of paper a colored pen, magazine, or pair of scissors lying around can suddenly become the source of inspiration that changes the course of the activity.

Fig 4.3: A higher level reached through internal ideas

Fig 4.4: A higher level reached through external inspiration in the form of affordances.

Additionally, the action identity can exclusively become higher level when the affordances correspond. When the initially aimless doodling progresses to a higher level ‘drawing a birthday card’, a piece of carton and envelope need to be obtained to fulfill its full action possibility. The easier to obtain those items, the less clear the cognitive idea of the action needs to be. If lying around already, the step to making a card can be easily made, even when still unsure whether it is going to work out (fig 4.5); if not in the house, the idea must be more clearly pictured and seem promising enough for a trip to the store (fig 4.6).

Fig 4.5: A sudden, ad-hoc, random idea is still tested if the action possibility is easy to create.

Fig 4.6: When creating the action possibility requires planning behavior, the idea needs to be more promising and predictable before it happens.

4.3. INTERNAL – EXTERNAL

The above indicates that stuff can enter a centre of self-organization in two ways. Either after being specifically searched for by the user themselves (which happens more easily when requiring less effort), or by presenting itself with its action possibilities and thus bringing the user to ideas for another or a higher level activity than the prepotent one.

In the former case, the process is started from an internal representation forming the external action possibilities through configuring stuff. In the latter, it is the external (physical) perception of an action possibility after which an internal (cognitive) idea is formed. These two directions, ‘searching’ and ‘stumbling upon’, continuously oscillate. While an action can be initiated through an action possibility that presents itself, it is mostly the act of specific searching that completes the stuff cell so it affords to perform the activity. A magazine on the table might inspire the user to ‘relax with a magazine’ (external to internal), but before the action can be performed, they will first search for something to sit on (internal to external), etc. In this process of self-organization the order parameter develops throughout the process. Either a wooden chair at the table or a lounge chair is found to ‘sit on’, both affording a higher level action of a different context. The next higher level completion of the stuff cell is done while searching for different affordances; a cup of coffee and a pen in the more serious setting of the wooden chair at the table, a footstool and animal paw slippers in the more relaxing context of the lounge chair.

Fig 4.7: The gradient between an action happening from external inspiration by a percieved action possibility and the specific searching for the creation of an action possibility. This gradient is the specificness of a ‘filter’, through which the surroundings are scanned for affordances. When the internal information is clear, the external information can be less clear and the activity will still happen (e.g. the artifact can be hidden in a drawer somewhere). When the external information is clear (i.e. the affordance is in plain sight) the internal information can be less defined and the activity can still happen.

As mapped out in fig 4.7 there is no clear cut between those two directions. On the contrary, they overlap when an affordance is searched for, but several possibilities are still open; the searching happens within a  certain category. As long as an artifact offers the pursued action possibility (e.g. ‘something to sit on’), one can still be inspired by other and higher levels the artifact affords (e.g. ‘sitting actively at a table’ or ‘sitting back and relaxing’).

Fig 4.8: The self-organizing process from non-activity to sitting in a lounge chair with a magazine and animal paw slippers. Over time, the activity grows more specific.

1 – Looking around in the category ‘something to do’

2 – Constructing a mental image of an affordance (reading a magazine, while sitting / e.g. “enjoying the afternoon”)

3 – Looking around in the category ‘something to sit on’

4 – Constructing a mental image of the affordance (reading a magazine while sitting comfortably / e.g. “cocooning”)

5 – Looking around specifically for animal paw slippers.

6 – Completion of the affordance.

The decision which artifact to use can be made exclusively through external information (“I am a new magazine, read me”), or exclusively through internal information (a specific screwdriver is searched for in a specific drawer tray), but mostly it is a combination of both. Tasks that are formed through an interplay between external and internal information, in which an initially takes more and more concrete form, are termed cognitively complex (Portugali & Stolk, 2014). The problem and solution co-evolve. The emergence of action identity and action possibilities in the form of configurations of stuff is such a task.

4.4  FROM LOCAL INTERACTION TO GLOBAL PATTERNS

4.4.1 Order and chaos

Stuff that is ordered by its properties is easy to specifically locate back, when known what to look for. In a library, the ultimate example of an ordered collection of stuff, a book can be found in minutes by searching for the first letter, then narrowing down to the second letter, the third letter etcetera. For this search not much external information is needed; a small indicator on the back of the book is enough. When systematically numbered, the item can, even in a closed storage system with thousands of books, be found quickly.

This type of ordered organization works efficiently, but only for planned behavior. It does not generate any ideas, which is why most libraries, for instance, offer displays with recently published or recommended books for visitors to stumble upon. It is the, to a certain extent, random placement of stuff that sparks coincidences and makes moments of contingency and serendipity possible.

4.4.2 The multi-scale gradient between order and chaos

As the self-organization of action and the self-organization of stuff happens through the constant alternation between searching and stumbling upon, both order and chaos is needed. Indeed, we can recognize the patterns of both states in almost every room.

The patterns of order and chaos are multi-scale; on a high level things can be ordered in categories, whereas zoomed in on those categories themselves, the items are not sorted any further; a kitchen cupboard is filled with cups and mugs (‘something to drink from’), but inside the cups are randomly placed. The category itself is hidden (behind a door) and will be specifically searched for, but when opened the cups individually present themselves by their visual information and inspire the user to pick a desired one. The other way around an ordered system can be split in subsystems that are more randomly spread around the house, such as when having multiple bookshelves (one for work-related, one for hobby-related and one for fictional books, and a smaller pile of newly bought ones), which give more life to a house than a clean and hidden archive.

The overall patterns to be found are multi-scale combinations of the soup-like and the grid-like – see fig 4.9 and 4.10.

Fig 4.9: Ordered chaos. Larger and slower scale: order. Smaller and faster scale: chaos

Fig 4.10: Chaotic order. Larger and slower scale: chaos. Smaller and faster scale: order

4.4.3 The gradient to self-organization

When we project the same method of combining patterns with different scale levels on the crystal-like pattern to be found around activities, we get outcomes that are equally recognizable. An example of ‘ordered self-organization’ (fig 4.11) are little compositions of stuff on shelves; one of chaotic self-organization’ (fig 4.12) are pans with different dishes on the table; and one of ‘self-organized order’ (fig 4.13) are small piles of sorted books around a working place. The processes of self-organization described above thus creates crystals from order and chaos in a leveled way, working its way from local to global.

Fig 4.11: Ordered self-organization. Larger and slower scale: order. Smaller and faster scale: self-organization

Fig 4.12: Chaotic self-organization: Larger and slower scale: chaos. Smaller and faster scale: self-organization

Fig 4.13: Self-organized order. Larger and slower scale: self-organization. Smaller and faster scale: chaos

Fig 4.14: Self-organized chaos. Larger and slower scale: self-organization. Smaller and faster scale: order