An Insistence on Looking
Imagine strolling through the city and running into a sidewalk robot that responds not only with functional precision but also empathetic understanding—a graceful dance of connection. This connection was at the heart of a recent theatrical exploration I conducted with makers Agat Sharma and Julia van der Putten.
How do people respond to robots on sidewalks or in other public spaces, and how should such robots respond in turn? In the past years, I have explored this question extensively in observations and graduation projects. Instrumental in these collaborations was our collaboration with the AMS institute, who provided access to a robot platform (Clearpath’s Husky) and living lab environment. We repeatedly found that people tried to engage in a continuous interaction with the robot, a back-and-forth. As of yet, however, our robots still commonly lack the empathic awareness, the affective responsiveness, and the readable expressivity to make them truly engage with this dynamic.
Interestingly, these are also questions that interest Agat Sharma and Julia van der Putten, both part of the collective Intimidation Tactics. For the graduation work of Agat at the research master DAS Theatre (AHK), they investigate how different kinds of agents can see and be seen in theatrical interactions—from the computer generated voice of David Attenborough, to a robot such as Husky.
To learn more from their approach to these questions, we thus embarked on a joint theatrical exploration, culminating in a 2-hour-long final theatrical experiment with audience. This builds on earlier design collaborations where I have drawn from the performing arts and the vocabulary they have developed for creating dynamic interactions. Instead of ‘using’ or ‘adopting’ methods as in these earlier efforts, however, the focus here was on immersing myself in the theatrical method and learn what value that brings in itself. This approach is more removed from the application context, but can provide more fundamental insights in the building blocks for investigating and developing appropriate interactions.
Making seeing resonate
In our first conversation, in a way that is much more associative and winding than I am used to, Julia and Agat sketch a theme of distance from our connected identity/nature and finding ways to craft connection. Their first association and intention with the Husky is to explore its ‘human’ identity, through the power of staging and framing. Ideas of breaking free from control, reminiscent of the most common AI tropes.
Yet when we actually take the Husky and explore what happens in interaction, something much richer emerges. We drive the robot manually, for safety and flexibility, as a kind of remote controlled puppet. Interestingly, the puppeteering of the robot turns out to work best when not focusing on the robot, but on who/what it is interacting with. This somehow shifts the interaction to one where there is resonance between people and the robot, a connection, an active relating to them in their humanness.
In our explorations, the movements I make with the robot gradually become more and more delicate. Enabled by the smooth floor in the theatre space, empowered by the emphasis on resonating with the people, the tiniest movement in resonance with the interaction becomes much more meaningful than the biggest movement would be by itself.
The performer interacting with the robot shares how it starts feeling as a friend providing support; feeling offended when I refer to the robot as an ‘it’, as it belies the connection they experienced. Discussions about our explorations quickly start to reference the Husky as an entity on its own, barely acknowledging my role as the puppeteer.
Between eyes and feeling seen
Our observations on the importance of resonance in building connection with the robot also connect to another theme within the work. Agat tells me about the importance of eyes and the rituals surrounding them. How the eyes are the last part given to a statue. How mountains and other natural objects can be given eyes, can be made into things that ‘see’ you. On what it means to be seen, or to be a seeing entity.
What do these eyes see when they look at me?
Within the theatrical experiment, this theme surfaced on numerous levels, also beyond ‘just’ the direct interaction between the mime performer and the robot. A camera was mounted on the robot, with its video being fed into a generative model to create visual reflections—which were in turn seen by and responded to the performer. On another layer, each of these parts was remote controlled by a different person; someone tuning the model, someone aiming the camera, and myself controlling the robot.
In all these layers, what we come back to is the fundamental role of closing the loop. Just putting eyes on the robot doesn’t do much; but treating them as if the robot is seeing from them in the puppeteering gives it a front-side to connect with. Likewise, the visual reflections generated become increasingly meaningful when the puppeteer starts interacting with them, or when the robot starts moving in response to them as well. In other words, seeing should be reflected in behavior to be seen.
Resonance as a method of inquiry
The explorations we conducted and the theatrical experiment they resulted in were incredibly rich. As the descriptions above hopefully suggest, there is a tacit component to the experience that touches something fundamental, but is hard to fully capture in words. Or, paraphrasing Agat, “each poem is a reflection of the whole of the poet”.
In this sense, embracing the theatrical method was antithetical to what I typically aim for in my academic work. Rather than a careful systematic search for generalizable knowledge, this was a much more actual—personal—search for what resonated. We could not really direct what these insights applied to; instead they organically emerged where things resonated. As such, my own background, in particular my prior experience with the performing arts as a hobby and this commitment to resonance over steering, heavily influenced what emerged from these explorations. And so too were the outcomes fundamentally influenced by the background of each of the others involved.
Practical implications
So, how do we derive actionable insights from these efforts? How can these abstract explorations in a theatre space and the reflections waxing poetic actually inform the design of behaviors for robots navigating our sidewalks and city?
Starting from the actual/personal, for me these explorations help further crystalize my ideas on resonance in interaction and robotics. The impression that, to establish true connection, we should go beyond just ‘having eyes’ and ‘sensing’ into seeing the other and responding to them. I found that this becomes resonance once those in the interaction mutually and open-mindedly adapt to each other, a back-and-forth with no preset outcome. From such resonance with the interaction, the tiniest nuances in the movement becomes much more meaningful than the biggest movement would be by itself. When resonance is achieved, the outcomes are necessarily shaped by the conditions and thus inherently reflect them.
Crucially, properly reflecting the conditions is precisely where current robots in our shared public spaces often break down. Examples include robots driving through a crime scene, blocking someone in a wheelchair, or being vandalized by curious kids. So what if, in these situations, a robot would instead establish a connection? Could we, perhaps, make robots really see the other and respond to them, using delicate nuances in movement to jointly find a suitable way out?
Most directly actionable in this regard is the idea of not just including eyes/cameras, but to instead insist on really looking: actively reflecting what is seen in adaptive reactions
Conclusion
This was a first exploration of the value that immersing ourselves in the performing arts could inform the design of behaviors for sidewalk robots and make it more responsible. It has raised many new questions, from how to actually implement such resonance, to how we might extract more generalizable behaviors from these kinds of theatrical explorations. But most importantly, they have manifested a swarm of ideas that might resonate with you.
Will we dare to design the behaviors of sidewalk robots in ways that do not make it feel like they are claiming our space, but instead like they are sharing it with us? Robots that insist on looking at us, really looking, when it matters.
