Discussion Session

Desiderata for Communication during Joint Action

Objectives and Motivating Questions 

An important area of social robotics is devoted to design robotic agents able to engage in joint action, which are broadly understood as any type of social interaction where two or more individuals coordinate their actions in pursuit of a common goal. In this sense, social robotics aim at developing robots able to establish interpersonal interactions with humans, so they can work together as a team to perform a shared objective. To carried out such joint activities, the participants must usefully coordinate each other’s actions which requires displaying different psychological devices for decoding intentions, goals or behavior. Although an important number of these devices depends on the interpreter’s capacity for generating predictions, participants often facilitate coordination by providing reliable implicit and explicit cues and establishing different communicative channels. This opens a myriad of important questions for social robotics: How is communication established between partners in the context of joint action? To what extent is plausible to draw conclusions from HHI to HRI in this case? What are the similarities and differences between communication in joint activities in contrast with other contexts (e.g. conversational contexts)? Can we take some lessons from other applications of Human-Robot communication to Joint action? 

Given these questions, the general aim of this session is to bring together different experiences in social robotics, philosophy and psychology with the objective of sketch some desiderata for a theory of communication in joint action for human-robot interaction. This general objective will be instantiated in several sub-objectives: 

1)    Drawing lessons from philosophy and psychology for communication during human robot interaction

2)    Discussing how certain application of human-robot communication can be applied to the case of joint action

3)    Establishing the central aspects of the communicative skills necessary for joint action 

4)    Enumerating and discussing a list of desiderata for communication in Joint Action for HRI. 

5)    Designing a future plan that will help us establish concrete objectives for overcoming the challenges imposed by communication in HRI. 

Scenario

To facilitate the discussion, we would like to build an illustrative examples that would serve as a common basis for discussion. It needs to be improved and discussed. However, we think it is crucial, to capture the subtlety of the work of each other and to build a common ground regarding the subject we want to discuss. The example aims at capturing the context of a collaborative task where partners may display the communicative channels and strategies at the focus of discussion.  

A human and a robot have the goal to assemble a piece of furniture. They should assemble the three parts (a, b, c) of the piece together using screws. The collective task requires the robot and the human to share a general goal (assemble the piece), deciding the order of sub-goals (Should we start assembling part a to b or c to A?), coordinate actions to perform certain sub-tasks (a subject must hold the part b while the other subject screw it into part c). Both agents have the pieces accessible in front of them and would participate to the task by assembling the piece of furniture. 

The initial state is the following:

• The piece of furniture has three parts which are in front of them
• The goal is putting all together
• Some pieces need to be held while they are screw into one another  
• There is no particular order to assemble the three pieces together but one can consider variations where they must start or finish with a particular piece

Actions available for each agent are the following:

• take an object on the table and hold it
• take an object and screw it into the other 
• give an object to the other agent

Each agent is able to infer the state of the world so it knows: 

• where each part is 
• if a part is reachable for itself
• if a part is reachable for the other one
• how the parts are correctly assembled

Moreover, we assume that each agent is able to observe the activity of the other. 

Guiding questions for the example: 

• What are the different strategies that agents can use for coordination? Indicating the next step (eye-gaze) or follow the heuristics ‘one starts, one infers the next step and follows’

• What are the different strategies that agents can use for deciding the different sub-goals? E.g. taking roles: leader/helper or verbal negotiation

• If they have to start holding part b, does the robot, who is closer to the part, starts or passing it to the human (leader)? 

Readings 

In order to make the panel more efficient, we would like to recommend to participants some readings that they may consider interesting and illuminating for the topic of the session: 

Plausible Desiderata: 

Mavridis, N. (2015). A review of verbal and non-verbal human–robot interactive communication. Robotics and Autonomous Systems, 63, 22-35.

Examples of Explicit and Implicit Human Robot Communication

 Jokinen, K. (2018). Dialogue Models for Socially Intelligent Robots. In International Conference on Social Robotics (pp. 127-138). Springer, Cham

 Ijuin, K. et al (2019) Eye-gaze in Social Robot Interactions – Grounding of Information and Eye-gaze Patterns. In The 33rd Annual Conference of the Japanese Society for Artificial Intelligence, 2019 

Human-Human Interaction (Philosophy and Psychology)  

Vesper, C. et al. (2017). Joint action: mental representations, shared information and general mechanisms for coordinating with others. Frontiers in Psychology, 7, 2039.

Butterfill, S. (2011). Joint action and development. The Philosophical Quarterly, 62(246), 23-47