blog - MUHAI

Uncommon Ground

2024-01-25T16:21:29+00:00

Robert Porzel, University of Bremen.

One major motivation of explainable artificial intelligence (XAI) is the desire to make the predictions of black-box machine learning (ML) models more transparent. Adadi and Berrada conducted a survey of the XAI literature and created an overview of common XAI methods - all of the methods presented range over the internal mechanisms of a ML model [Adadi and Berrada, 2018], which can be called introspective explanations. More recently, there has been an increasing need to explain AI behavior to non-expert users of AI systems. Therefore, XAI needs to focus more on end-users as the recipients of the explanations and on how XAI methods can be evaluated from a human-centered point of view. This sheds a different light on estimating the quality of an explanation and how it can be tailored adequately to the given user. This view is supported by the argument that explanation should be understood as interactive conversations. The usefulness of explanations can depend on the context in which they are given, including the situation and the user. In domains such as everyday activities, human users interact with AI agents repeatedly over time. When the two disagree, helpful and succinct explanations of the AI’s behavior should be tailored to the user based on its memory of their previous interactions.

To foster a human-centered approach to XAI, individuals and their preferences should be considered by appropriate user models. These can encompass, e.g., a level of education, cultural background, familiarity and interest in technology, perception of the agent, mood, and social settings. To accommodate these factors, personalization of AI explanations might be crucial. This is particularly relevant in domains where users interact with the same artificial agent over a long period, such as smart home devices, robots in household and care domains, and other personal AI applications. If users ask for explanations because they expected different AI behaviour, there must be some disagreement between users and AI in their respective beliefs and knowledge, which is what we call their uncommon ground. Explanations should allow the user to realize where the uncommon ground is. In error cases, this can empower users to correct the AI’s behaviour for future interactions; in other situations, it can enable them to see that they are missing important knowledge themselves. To facilitate this, explanations by the AI should reflect its experience from previous interactions with the same user. Personalization can improve user satisfaction by using user models created with direct or indirect user input, i.e., user entry or automated systems that adapt to user behaviour. Although direct user inputs can have advantages, an agent that is not properly configured may be less acceptable, thus personalization through indirect adaptation might be preferential. We argue that it is thus necessary to consider how to generate extrospective explanations: explanations that are given with respect not only to the system itself but also take into account what the system knows about the user’s expectations and draw on experience from earlier interactions. This extrospective perspective on XAI is at the heart of our work, as we focus on domains where users interact with smart software, intelligent devices, and autonomous agents/robots repeatedly over time, e.g., smart home device and household. This, in a sense, entails that users have their ‘own’ personalized AI. Explanations in the context of repeated interactions have not been studied as much. Therefore, we seek to examine this challenging problem systematically. There is a growing awareness in the XAI community that explanations ought to be considered and evaluated in the context of the user for whom they are intended. Previous works have pointed out the problem of implied-but-not-stated contrastive/counterfactual cases when humans ask for explanations as well as that there are many situations in which the user’s goal in asking for an explanation relies on understanding why the AI prediction differed from their expectation.

We claim that in the domain of AI in the home (e.g., smart home devices, household robots), extrospective explanations can provide more helpful information for end-users in a more human-centered way, enabling them to understand the reason for unexpected agent behaviour. We aim to achieve this by focusing on explanations for transparent, reasoning-based AI systems and explicitly modelling user and discourse context in the agent’s knowledge. If the AI predicts the same out- come based on the same situational knowledge and ontology in two equivalent situations, the respective explanations could differ based on the user asking for them: For each user, the AI would output only a part of their reasoning. Specifically, the AI should output those facts or inferences that it expects are most likely to surprise the user. This should be whatever fact or logical inference the AI has the least reason to believe that the user agrees with or is aware of. In other domains, however, there might be different reasons why explanations should be purely introspective. There are many areas of life where AI is or will be used, in which it will be important that explanations are consistent across different users or that they are a complete representation of the AI reasoning (e.g., medical or legal problems). Therefore, it will be essential to distinguish between explanations aimed at helping individual users understand local, surprising AI predictions and explanations aimed at increasing ML systems’ transparency.

[Adadi and Berrada, 2018] Amina Adadi and Mohammed Berrada. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6:52138–52160, 2018.

Full Paper in the papers page:

Laura Spillner, Nima Zargham, Mihai Pomarlan, Robert Porzel and Rainer Malaka. Finding Uncommon Ground: A Human-Centered Model for Extrospective Explanations. In Proceedings of the 2023 IJCAI workshop on Explainable Artificial Intelligence (XAI). 2023. BibTeX, pdf

Do you speak AI?

2023-03-30T08:17:40+00:00

Interview with Katrien Beuls, UNamur.

We are pleased to announce that MUHAI researcher Katrien Beuls is now Lecturer in Artificial Intelligence at the University of Namur (Belgium) and we are happy to welcome UNamur in the MUHAI consortium! Read her interview for the University of Namur's magazine "Omalius".

Katrien Beuls is undoubtedly a fine example of the growing number of women in STEM (science, technology, engineering, and mathematics) careers. After a rather literary career, guided by her curiosity, she began studying computer science and became interested in computational methods for processing human language with the help of Artificial Intelligence (AI).

After an atypical curriculum, her meeting with Luc Steels (VUB), a pioneer of artificial intelligence in Europe and an expert in evolutionary computational linguistics, and an international experience as a student and post-doctoral researcher, Katrien Beuls has just moved to UNamur.

"A position as 'Lecturer in Artificial Intelligence' was vacant. I applied and here I am! I found it interesting that the job description did not restrict the function to a specific sub-field of AI. I find UNamur extraordinary, because of the human size of the institution, which allows for a lot of interaction quickly, as you get to know everyone quickly! I also had the impression that there was a certain open-mindedness in the Faculty of Computer Science and that they were interested in new methodologies. I really feel that UNamur is an excellent place to create my own interdisciplinary research team.

A passionate researcher
"I am trying to build intelligent systems, so-called cognitive agents that implement the 'perceive', 'reason', 'act' cycle that constitutes the decision process. These intelligent agents have to solve communication tasks in teams. The aim is to build a common language to refer to a situation, objects, etc. This is done in the form of a game according to an established protocol. An agent chooses a topic of conversation and has to ask himself the question: what do I have to say so that the other knows what I am talking about? They have perceptual abilities but see the world or segments of it without labels and have no language system.

This is a bit like how a child learns a language. At the beginning, he has no vocabulary and does not know grammar. They have to learn to relate what they see to the word or concept that defines it. This is a process that involves many cognitive abilities, at the sensorimotor, conceptual and linguistic levels. Human beings are the only ones who can express themselves through a constructed language. This is a great source of inspiration.

New dynamic systems: learning and adaptating
Let's take a first game. Several agents are faced with the same scene (see opposite). The first agent wants to point to the red teddy bear and calls it 'it'. But multiple factors are linked to this information: the location in space, the material, the colour... A mechanism will store the information and classify it. The term "it" is too vague, there are several possible interpretations. Is it the plush? Its colour? To solve this chaos, another mechanism is introduced to manage the problems of association between shape and meaning: a score between 0 and 1 is added (1 means a good association, 0 a bad one). As interactions with multiple agents take place, the scores will increase or decrease. This makes it possible to verify the correct understanding and the choice of a precise common term. But we are only dealing here with basic lexical terms: vocabulary words. We also need to introduce grammar.

In order to move towards more complex structures on a semantic and conceptual level, we will have to push the experiment further. The red plush toy is already a complex concept: its material (furry fabric), its shape (small monster), its colour (in RGB form), its location in space (to the left, next to, etc.), its size (small, large), etc. All these concepts will have to be isolated. Compare red objects, for example, in order to be sure that we are talking about colour. In this way, we can refine our understanding until a common term is identified and accepted.

"The AI method for the language game I am working on involves a multitude of agents that will introduce many ideas and concepts in order to finally find a common language and thus a robust, adaptive and productive communication protocol."

Language: a layer of abstraction about the world and its understanding
Let's go back to humans and two of the fundamental capacities in language acquisition: reading intention and finding patterns. According to the theory of construction grammar, all of a language user's knowledge is composed of constructions, i.e. associations between form and meaning.

"Take a young child who is asked, 'Would you like some more milk? Depending on his reaction, he may or may not get any. It is from this interaction that the child can make a hypothesis about the meaning of the sentence he or she has heard: he or she "reads" the intention of the speaker in the given context. By dint of repetition, the child consolidates this question, the compositional structure of which he is still unable to understand. He also adapts over time. Do you want some more water? And the process repeats itself. Thanks to the pattern-seeking mechanism, he will come to the following conclusion: Do you want X? I get X."

Intent reading is complicated to implement when it comes to AI because it refers to the ability to hypothesize the intended meaning of an observed utterance based on the situational context in which it is uttered. Pattern matching refers to the ability to generalize about pairs of observed statements and their hypothesized meaning, producing form-meaning mappings of varying degrees of abstraction.

We create algorithms starting from an inventory of basic operations to develop programs that will allow the construction of conceptual structures in an intent reading mode. For an agent, the process will be as follows to answer a question such as "How many blue cubes are there in the image?" Colour alone is a complex concept as a robot reads it in RGB mode. Green can be mistaken for blue without checking that the nuances have been correctly understood.

By dint of repetition and interaction with other agents, and with a precise goal, it will consolidate the different concepts it has encountered by calling on all its memory and associating the elements with each other. This will create a holistic expression in which it can construct meaning.

Deliberative intelligence
The book "Thinking, fast and slow", written in 2011 by psychologist Daniel Kahneman explains the dichotomy between two modes of thinking: 'System 1' is fast, instinctive and emotional and 'System 2' is slower, more deliberative and logical. System 1 is basic. System 2 is reflective and reasoned.

"The AI world is currently based on a reactive intelligence system. I am trying to implement deliberative intelligence. One of the major limitations of most current AI systems is that they learn to solve a single task based on millions of examples. This is not intelligence. Having segmented knowledge is absurd! Intelligence is about solving new problems and adapting to the knowledge you already have.

In addition to teaching and supervising ongoing research projects, Katrien regularly organises interdisciplinary workshops with linguists, psychologists and biologists for inspiration.

--

"I like working in a team. It takes several people to learn! It is us as individuals interacting with each other who have ended up building our language. In fact, I am like my agents." Katrien Beuls

This interview was conducted for the "Expert" section of the University of Namur's Omalius magazine #27 (December 2022).

Photo of Katrien Beuls - Credits: © Christophe Danaux

Pragmatics: the secret ingredient

2023-03-06T10:29:33+00:00

Anna Morbiato, VIU.

Imagine you're at a house party. You've just met someone new and you want to make a good impression. So, you ask them "How's it going?". To a machine, this phrase might seem like a simple question, but to a human it has a whole range of meanings. It could be a polite greeting, a genuine inquiry about how the person is doing, or even a subtle way of asking them to leave. The machine might not be able to pick up on the nuances of the conversation, but a human would be able to decipher the true meaning behind the words. That's pragmatics in action!

Pragmatics is the study of how we use language in context. It looks at how the meaning of words and expressions may change depending on the situation and the people involved. It also considers how our intentions and our social relationships affect the way we communicate. It is like a party trick - it's the art of knowing how to communicate the right thing at the right time!

Pragmatics is the focus of the paper Pragmatics of Narration with Language, which is part of the MUHAI white paper Foundations for Meaning and Understanding in Human-centric AI. Specifically, the paper looks at how research in pragmatics and cognition over the past 50 years has helped us to better understand natural languages, narratives, and language processing. The paper explores how factors such as the speaker's communicative aims, perceptual experiences, stance/viewpoint, as well as cultural and social aspects of communication, can affect how humans use language.

One of the aspects the paper looks at is coreference disambiguation, which is essential for parsers and machine learning when it comes to understanding natural language. Imagine you're making pizza with a friend; she is kneading the dough and you want to check on it. So, you ask "How's it going?". To a machine, this phrase might seem like a simple question, but if the parser incorrectly disambiguates the pronoun "it" as referring to the friend, instead of the dish, it could lead to confusion. The machine might interpret the question as asking about the friend's wellbeing, instead of the progress of the dish. Coreference disambiguation is the process of determining which words or phrases in a sentence refer to the same entity. This helps the parser to accurately interpret the sentence and understand the intended meaning.

Another both interesting and challenging realm is that of speech acts. Speech acts are the ways in which we use language to express our intentions and convey subtle or implicit meaning, such as orders, suggestions, promises, and requests, as well as a variety of other communicative goals. Getting back to your cooking session with your friend, imagine your pizza is baking in the oven, and your friend wants to check on it. So, she suggests: "Maybe it's time to take it out now?". To a machine, this phrase might seem like a simple suggestion, but you would instead be able to detect whether it is a nice way of hinting that the pizza is burning (and you should have already turned the oven off). Pragmatics is like a chef's special ingredient - it's the art of knowing how to season a dish just right so it tastes delicious!

That's why pragmatics is so important, both for humans and for parsers: because it helps to make sense of the context of conversations and interactions. The goal is the development of deep learning models that can understand natural language also taking into account these nuances of human interaction. This technology is still in its infancy, but it shows great promise for improving the accuracy of machine learning models in interpreting natural language.

Credits
Intro image - photo by Andrea Piacquadio via Pexels

Deconstructing Recipes

2022-10-03T12:20:05+00:00

Anna Morbiato and Patrizia Cani, VIU.

What is the secret ingredient of recipes? In recipes, we talk about ingredients, sometimes many of them, which get moved into different containers and transformed in a thousand different ways, thus turning into other things (such as a dough, or a puree). Moreover, these ingredients and ‘resultant objects’ are often unmentioned, as in ‘Bake until crispy and golden.’ ‘Pour until saturated’. Yet, we are always capable of understanding what that specific step or instruction is talking about. How? This is what we aimed at exploring with our paper, "Deconstructing Recipes: A constructionist comparative analysis of result and cohesion".

Recipes are goal-oriented technical texts which present unique linguistic features: they describe sets of actions carried out in a specific order in time; entities that are mentioned at the beginning (ingredients) get moved and transformed; each step results in different items (in linguistics we call these ‘resultant objects’), which in turn get again moved and transformed. Agents are always unmentioned, and patients – namely ingredients and resultant objects – often so (in linguistics, this is called ‘zero anaphora’ and ‘co-referential deletion’).

Aware of the unique technical nature of recipes texts and of the wide occurrence of co-referential deletion, our first research question concerns the way readers correctly decode what recipes talk about in each step (coreference disambiguation and cohesion). The second question has to do with the functions fulfilled by recipe texts. In a comparative cross-linguistic view, we sought to determine what forms are used to encode functions of giving instructions, temporal sequence, resultant location, object and state. We did this by comparing three languages: English, Italian, and Chinese. In particular, what are the cohesive devices each language employs to achieve textual cohesion and how does each language employ them? The approach we chose is that of construction grammar, which sees language as a collection of form-meaning pairs: not only does the lexicon convey meaning, but constructions do so, too. Our corpus was composed of user-created recipe texts describing the same dish, pumpkin bread rolls, drawn from international food blogs. The register is casual and texts have not gone through a process of editing. We chose authentic texts over translations of the same recipe, as they may be linguistically biased and reflect the structure of the original text.

Among the most fascinating aspects we looked at is reference tracking and anaphora. English presents a good amount of Full Nps repetition and synonyms, but co-referential deletion remains the most important device being employed. On the other hand, English presents a very limited use of pronouns and verbal agreement, which are the main devices used in the Italian recipe. The Chinese language, finally, uses almost exclusively zero anaphors. Sometimes, in recipes, reference can even be missing: in some cases, referents corresponding to resultant objects, including the final result of the recipe, were never mentioned. But how is the reader able to understand exactly the contents of the instructions, when entities are not mentioned or if anaphoric means may refer back to more than one referent? The answer lies in many non-linguistic cues that inherently characterise our cognitive abilities to understand texts. This may be the pictures matching the instructions: recipes are multimodal texts where pictures play a big role in disambiguation and contextualization. Sometimes, however, inference and world knowledge play a vital role: referents can be traced only through context and inferential mechanisms. Crucially, while we humans have little difficulty in doing so, even the most advanced linguistic parser may not be able to disambiguate anaphors.

This article contains excerpts and figures from the preprint versions of the following works:

Deconstructing recipes: a constructionist comparative analysis of result and cohesion in Chinese, English and Italian

If you are interested in the topic, you may consult Patrizia’s MA thesis at this link:

http://dspace.unive.it/handle/10579/20730

Credits
Photo by congerdesign on Pixabay

Foundations for Meaning and Understanding in Human-centric AI

2022-07-26T16:31:01+00:00

Carlo Santagiustina, VIU.

The summary that follows contains excerpts from the open-access volume of the MUHAI Deliverable 1.1:
Steels, Luc (ed.). (2022). Foundations for Meaning and Understanding in Human-centric AI. In Foundations for Meaning and Understanding in Human-centric AI (1-6-2022, p. 152) [Computer software]. Venice International University. https://doi.org/10.5281/zenodo.6666820

Through the Foundations for Meaning and Understanding in Human-centric AI, the MUHAI project offers an in-depth and integrated overview of narratives and understanding in different disciplines and research fields.

The volume builds upon recent insights and findings from social and cognitive sciences, humanities and other fields for which narratives have been found to play a relevant role in human understanding and decision-making processes. This, to map the state-of-the-art of narrative-centric studies and to identify the most promising research streams for tomorrow’s AI. In particular:

Ch. 1 - Towards Meaningful Human-Centric AI, focuses on the conceptual foundations of human-centric AI and discusses some fundamental questions related to the proposed approach. Such as, what is the nature of meaning and understanding? And why is understanding needed to make AI more transparent and intelligible to humans?
Ch. 2 - From Narrative Economics to Economists’ Narratives, explores the role of narratives in economic and social affairs, focusing on their usage for uncertainty avoidance, decision-making and decision justification. This chapter also highlights the strategic aspects of narratives and their relation to cognitive biases affecting humans’ sense- and decision-making processes.
Ch. 3 - Narratives in Historical Sciences, highlights how Historical sciences try to offer causal explanations for non-recurrent phenomena, typically using incomplete and fragmentary evidence from the past. Through a case study on the French Revolution (1789–1799), this chapter shows in which terms narrative explanations go beyond mere description and have the potential for empirical testing.
Ch. 4 - Clinical Narratives for Causal Understanding in Medicine, views clinical trials as causal narratives in the biomedical domain and as a means to understand the causal mechanisms of treatment effects. Through a use-case, this chapter describes what narratives in this domain are, and how they are formed and tested. This contribution also discusses how narratives can be represented computationally and how AI techniques can support domain experts in the generation of new hypotheses.
Ch. 5 - Narratives in social neuroscience, suggests that narratives are a basic component of human cognition. Narratives are found to be a tool used by the human brain to make sense of experience and to build representations of the world and of ourselves. This chapter also presents studies about the area of moral values that show that the notion of narratives must take a central role in the future of social neuroscience and related fields.
Ch. 6 - Narrative Art Interpretation, clarifies the narrative view on art interpretation with a concrete example of a painting by Caravaggio and it explores the implications of this view for building theories and mechanisms for dealing with meaning and understanding in AI systems.
Ch. 7 - Pragmatics of Narration with Language, explores how research conducted into pragmatics and cognition can deepen our knowledge of natural languages, narratives, and ultimately language processing. The paper also illustrates how some of the fundamental pragmatic elements that characterise narratives heavily influence the ways languages structure basic linguistic elements, such as clauses, sentences, and texts.

As the conclusion of this volume highlights, narratives are ubiquitous and essential cognitive goods used in key spheres of our professional and social life, including the socio-economic, artistic, and scientific domain.

Through this work, the MUHAI consortium has undertaken a first (but far-reaching) step towards meaningful AI, proposing a variety of R&D paths to be explored, implemented, and tested in the next phases of the project. This kind of AI integrates and goes beyond ML and statistical methods for pattern recognition, completion and prediction, and explores how narrative-centric methods can inform the next generation of AI researchers and help them integrate in their systems narrative-related aspects of humans’ individual and collective understanding, which have yet to be fully acknowledged in AI research.

Our explorations have yielded a wealth of insights and possible applications of meaningful AI in a diverse set of fields, ranging from the analysis of debates about social inequality to hypothesis generation in scientific research.

By acknowledging the prominent role of narratives in understanding, the volume tries to facilitate the development of new methods that can better complement human understanding processes, eventually helping us identify and mitigate some cognitive biases that relate to narrative fallacies. Meaningful AI methods should be designed to fit to real world situations where inputs are typically sparse, fragmentary, ambiguous, underspecified, uncertain, vague, occasionally contradictory, and possibly deliberately biased, for example, because the producer of inputs is trying to deceive or manipulate. For understanding inputs with the aforementioned characteristics, different solution paths may have to be considered at the risk of exploding combinatorial complexity.

Understanding may be hard for AI systems because of the hermeneutic paradox: to understand the whole, you need to understand the parts, but to understand the parts, you need to understand the whole. This calls for integrated control (and meta control) structures other than a simple linear flow of processes organised in an automated pipeline.

As this volume suggests, meaningful AI models should be inspired by recent works in the field of knowledge representation, knowledge-based systems, fine-grained language processing and semantic web technologies, as well as other non-AI research streams explored in this volume, that we recommend you to read.

Foundations for Meaning and Understanding in Human-centric AI can be downloaded at this link: https://doi.org/10.5281/zenodo.6666820

The FCG Editor: a new milestone for linguistics and human-centric AI

2022-06-28T08:35:13+00:00

Remi van Trijp, CSL.

The Fluid Construction Grammar (FCG) Editor is a free and innovative Integrated Development Environment (IDE) for engineering computational construction grammars. The design philosophy and description of the FCG Editor has now appeared in a PlosOne Article, including an introduction to the field of (computational) construction grammar.

When people hear the word "grammar", most of them still think about a set of syntactic rules to combine words (and their concepts) in a compositional fashion. Most NLP (Natural Language Processing) systems therefore consider grammar to be equal to syntactic parsing, so syntax is simply one of the components of a traditional pipeline that can be useful for downstream tasks. In the MUHAI project, we take a different approach inspired by cognitive-functional linguistics, in which grammar itself is meaningful: it expresses how people conceptualise reality, and which perspective they take on the events that they perceive in their daily lives. The most suited linguistic theory for this approach is Construction Grammar, in which all of linguistic knowledge - including grammar - can be modeled as mappings between meaning and form, which are called constructions. One of the core components of MUHAI technologies is therefore Fluid Construction Grammar, the world's most advanced platform for implementing computational construction grammars, co-developed by the MUHAI partners Sony CSL Paris and the VUB Artificial Intelligence Laboratory.

While computational construction grammar has the potential to lead to fundamental breakthroughs in Natural Language Understanding, it turns out that it is very difficult to develop models of constructional language processing, partly because user-friendly developer tools were missing. This is why our MUHAI partners from Paris and Brussels, along with the KU Leuven and the University of Namur have released the FCG Editor: a free and innovative Integrated Development Environment (IDE) for engineering computational construction grammars using the Fluid Construction Grammar (FCG) formalism.

"With Fluid Construction Grammar, we have an amazing tool for implementing new kinds of language models, but you need to have significant software engineering skills to use it. So when we gave tutorials on FCG, we found that this technical threshold was a demotivating factor for many participants," says Dr Remi van Trijp, research leader of the Language Team at the Sony Computer Science Laboratories Paris. "We therefore worked on a first prototype that would offer people a more user-friendly experience, and then proceeded to test and refine the FCG Editor by letting NLP and AI students use it for implementing constructional language models." The result of this hands-on approach is an innovative IDE for Fluid Construction Grammar that strikes a unique balance between user-friendliness and open-mindedness, drawing inspiration from the field of interactive programming.

The FCG editor provides all of the editing possibilities that you can expect from an IDE, and also allows interactive programming through a listener (similar to command-line interfaces), an interactive web interface that allows the user to inspect every detail of processing, configuration helpers, and a "construction wizard" that assists users in the definition of linguistic constructions. The main target audience for the FCG Editor is linguists, particularly construction grammarians, who wish to operationalize and test their theories; but also computational linguists and AI researchers can use the FCG Editor to develop models of constructional language processing to enhance their language technologies. "I believe the FCG Editor is an important milestone that can finally boost the development of computational construction grammars," concludes van Trijp.

The FCG Editor now features in a PlosOne Article that also includes an introduction to Construction Grammar and computational construction grammars. Besides funding from the Sony Computer Science Laboratories Paris and MUHAI, additional funding came from the European Union's Horizon 2020 research and innovation programme AI4EEU under grant agreement No 825619, as well as the Flanders Research Foundation.

Related Links
FCG Editor
PlosOne Paper
MUHAI papers

Acknowledgements
Original photo by Jan Baborák on Unsplash

Linguistic Alignment for Chatbots

2022-03-23T17:39:00+00:00

Laura Spillner, UHB.

Over the last decade, conversational agents have slowly but surely become very common in our daily lives. Conversational agents - these are computer agents, robots or software which can understand and talk to a user in natural language - include assistants on our smartphones like Siri or Google Assistant, voice-controlled smart home devices like Google Home and Amazon Alexa, as well as online chatbots like the ones frequently used in customer service. Clearly, natural language plays an important role when it comes to fluent and easy interaction with computers and mobile devices. But in spite of their popularity and the considerable amount of recent research in AI-based understanding and generation of natural language, these interfaces still face some difficult challenges.

Human communication comes with a number of peculiarities: conversations are often informal and use non-standard language compared to formal written texts; ignoring grammar and spelling rules. And even when writing chat messages, many people employ the same kind of informal language as they do when talking in person. New types of expressions such as emojis and their often variable meaning can also hinder successful human-computer communication.

Many researchers have shown that users tend to react socially to computer agents, meaning that they interact with computers or robots in the same way as they do with other humans [1]. This is the case especially if these agents also employ natural language, or if they imitate humans' social and affective cues [2, 3]. But when it comes to conversational agents, this can easily lead to misunderstandings or communication breakdowns. As many users now, even popular chatbots and voice assistants still fail quite often in real-world usage: They misunderstand what we're trying to ask or what we want it to do, they miss important context, and their answers are often robotic and unnatural.

In the field of psycho-linguistics, researchers have studied how two people engaged in a conversation are able to achieve mutual understanding and prevent communicative failures. The social and collaborative nature of human conversation is well established, in theories such as Communication accommodation theory [4] and Interactive alignment theory [5]. Based on these theories, successful communication between humans depends on both participants’ ability to adapt to the language of their conversation partner. This means that two speakers imitate each other’s use of language in many ways, without being aware of that they are doing it. They repeat each others' word choices, phrasings, and sentence structure, and slowly become more similar to one another in their overall language style. In this way, they are also able to achieve a more similar understanding of the situation under discussion. Thus, imitating one's conversation partner effectively paves the way for successful communication.

In human interaction, this kind of "linguistic alignment" is what makes successful communication possible. Aside from language itself, it has also been shown that people imitate others when they interact with them, mirroring their facial expressions and gestures. The same effect happens when people imitate each other’s language style in conversation. This adaptation or imitation directly impacts how positively someone is perceived by others: If someone aligns more strongly to their conversation partner, their partner will think better of them. Moreover, alignment is not just important in terms of perception, but also directly impacts how well people are able to work together. For example, when it comes to solving tasks together with others, people who's language aligns more closely to each other are more successful in their tasks [6]. They also report lower workload and higher engagement during the task, when they align better with their partner [7].

Of course, our interaction with chatbots and other conversational agents is very much impacted by how we chat with other people. Chat apps are among the most popular mobile applications and the most popular types of social media; and chat conversations have evolved their own kind of language too. This kind of language is less adherent to classic dialog structure and formal grammar or spelling rules. All of this makes natural conversations difficult for AI. Therefore, we reasoned that considering the psychological model of human communication will be essential in order to improve interaction with conversational agents. If users interact with chatbots as if they were social agents, in the same way in which they interact with other people, then they might subconsciously expect them to adhere to the same communication patterns as other people would.

Based on this idea, we developed a new chatbot called KONRAD. Konrad is able to produce a linguistic alignment effect, similar to the alignment that happens in human conversations. In order to do this, the chatbot repeats the user’s word choices and uses similar similar sentence structures as the user. Konrad works as a simple goal-oriented chatbot, which intends to help users decide which movie to watch at a local cinema. Conversations with Konrad are user-initiative and based on the intent action structure: This means that there is no chit-chat or small talk conversation as there is with other AI chatbots like Cleverbot. Instead users have the initiative and ask the chatbot questions, similar to Siri and other smart assistants. In this case, users can ask about which movies are currently running, information about those movies, and general information about the cinema. The chatbot then predicts the intent of the user’s question (one of a number of pre-defined categories, such as the starting time of a specific movie). It looks up the information needed to answer the question, and then formulates an answer. Using Konrad, we conducted an online study to investigate how users would perceive and evaluate a chatbot that mimics human behavior through linguistic alignment.

We expected that, based on the theory that people treat conversational agents and robots as social agents, the study participants would align their language more strongly to the chatbot if it also exhibited an alignment effect. Moreover, solving tasks and acquiring information are key use cases of chatbots and other conversational agents; and previous studies have shown that alignment reduces workload when solving tasks together. Thus, we also hypothesized that the participant’s workload when solving a task with the help of the chatbot should be lower if the chatbot aligns to them linguistically.

We built three different versions of Konrad: The first version does not have any alignment, but instead works in the same way as most conversational agents and chatbots currently do; by using pre-defined templates for answers. This is comparable to other popular conversational agents such as Siri, as well as state-of-the-art chatbot frameworks like Google’s Dialogflow. They generally build their answers to user requests in pre-defined ways, inserting information into a template answer.
The other two versions of our chatbot Konrad implement two different variants of the desired alignment effect. The first one only implements a basic alignment effect, in which the chatbot adapts to the user's choice of words: Normally, the chatbot uses pre-defined answers. But when the chatbot recognizes that the user prefers a specific term for something, a word that is a synonym to the one the chatbot would otherwise use, then it replaces its pre-defined term with that synonym. In this way, the answers are still similar to the pre-defined template ones, only with some of the words replaced. The second alignment version does away with the template answers all-together. In this version, we aimed at producing a structural alignment effect, i.e. imitating the syntactic structure of the user’s question in the chatbot’s answer. In order to achieve this, the chatbot dynamically generates its answer using the structure of the question as a basis - it rearranges the question into an answer format, and then inserts the information necessary to answer the question at the appropriate position. For example, if someone asks Konrad “Which actor is Luke Skywalker played by?” it would answer “Luke Skywalker is played by Mark Hamill”. But if you ask it “Who portrays Luke Skywalker in Star Wars?” it answers “Mark Hamill portrays Luke Skywalker in Star Wars”. This way, it keeps the syntactic elements from the question (here, the passive vs. active voice) and also automatically reuses the terms which the user prefers (here, "plays" vs "portrays") without needing to identify possible synonyms.

In our study, the chatbot Konrad was tested by 75 participants. They could access it online, either on their computer or on a smartphone. Each participant was randomly assigned one of the three versions of the chatbot, and only interacted with this one version. They were tasked with using the chatbot to find a movie that they would want to see. During the conversation, they could ask the chatbot about when which movie was running, ask it for details about the different movies, or to give them reviews and recommendations. Once they had decided on a movie, they were asked to fill in a short online survey: with this survey, we assessed their general opinion of the chatbot, and also measured their perceived workload and engagement during the conversation. Afterwards, we evaluated both the information from the survey, as well as the conversations itself. By analyzing the conversations that users had with the chatbot, we were able to measure how strongly either of them aligned to the other one.

Our study supported our hypothesis that linguistic alignment would impact the interaction with the chatbot. Firstly, we found that user alignment was correlated with chatbot alignment, showing that users did in fact react to the alignment effect displayed by Konrad by aligning more strongly to it in turn. Secondly, the users' workload was lower and engagement was higher when they tested the variants of the chatbot with added alignment. In fact, workload was even negatively correlated with alignment, meaning that those which had higher alignment throughout the conversation experienced lower workload in solving the task at hand. The same has been shown in previous work for human conversation [7] - our results show that this connection between alignment and task workload applies to human-computer-communication as well.

In this study, we tested Konrad in the film domain, because many people are familiar with this domain and there is not a huge influence of prior knowledge. However, alignment might differ depending on the domain as well as the user's motivation. For example, the effect of alignment might be different when searching for information on an unfamiliar topic: you might not want the chatbot to imitate your choice of words, if you are not certain which terms are correct. Moreover, we wonder whether there is a difference between solving a task with the chatbot, compared to engaging in small talk. Finally, the length of the sentences and the conversation might have an influence on the alignment, which we did not investigate here. These topics we plan to look into in future studies. We expect that additional strategies and implementations of alignment will be required in order to extend this idea to a more general approach - for Konrad, we relied on a combination of neural networks (in order to understand the topic of the user's question) and grammatical rules (in order to construct the chatbot's answers). In future implementations however, other usage domains and more general conversations will likely require more flexible approaches.

The results of this study showed that the phenomenon of linguistic alignment has an impact on conversations between humans and computers. Thus, it should be taken into account when developing natural language-based interfaces such as chatbots. Alignment plays an important role in achieving successful communication between humans, and implementing it in conversational agents can clearly have real-world benefits for user interaction.

References
[1] Clifford Nass and Youngme Moon. 2000. Machines and mindlessness: Social responses to computers. Journal of social issues 56, 1 (2000), 81–103.
[2] Theo Araujo. 2018. Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conver- sational agent and company perceptions. Computers in Human Behavior 85 (2018), 183–189.
[3] Eun Go and S Shyam Sundar. 2019. Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions. Computers in Human Behavior 97 (2019), 304–316.
[4] Howard Giles and Peter Powesland. 1997. Accommodation theory. In Sociolinguistics. Springer, 232–239.
[5] Martin J Pickering and Simon Garrod. 2004. Toward a mechanistic psychology of dialogue. Behavioral and brain sciences 27, 2 (2004), 169–190.
[6] David Reitter and Johanna D. Moore. 2014. Alignment and task success in spoken dialogue. Journal of Memory and Language 76 (2014), 29–46.
[7] Paul Thomas, Mary Czerwinski, Daniel McDuff, Nick Craswell, and Glo- ria Mark. 2018. Style and alignment in information-seeking conversation. In Proceedings of the 2018 Conference on Human Information Interaction and Re- trieval. 42–51.

This article contains excerpts and figures from a previously published paper: Laura Spillner and Nina Wenig. 2021. Talk to Me on My Level – Linguistic Alignment for Chatbots. In Proceedings of the 23rd international conference on mobile human-computer interaction. New York, USA: Association for Comput- ing Machinery.

Credits
Intro photo by Pavel Danilyuk from Pexels

Framing reality

2022-01-28T16:22:23+00:00

Remi van Trijp and Martina Galletti, CSL.

One of the reasons why it is so difficult to develop human-centric AI systems is that such systems need to "understand" the world and human activities in a way that is compatible with how humans make sense of the world. The crux of the matter is that each person has their own unique way of doing so: reality is so mind-bogglingly complex that we constantly need to make choices about which information is relevant, and which elements of a situation should be highlighted or obscured. This process – in which a person puts a situation in a particular perspective to express their beliefs, desires, and intentions – is called "framing".

A “frame” is a structured piece of knowledge that we build up and maintain through experience. At its most basic level, a frame can be considered as a template of a scene with several open roles (called “Frame Elements”) that need to be filled in.

Figure 1 illustrates a scene that we may perceive, and everyone who has ever prepared a meal in a kitchen can immediately “frame” the scene. There are multiple frames possible, such as “Cooking”, but here we decided to frame the woman’s activity as a “Baking” event. The Baking frame includes several Frame Elements such as the person who does the baking, the food that is being prepared, utensils for doing so, a time and place (usually the kitchen), and so on. We can then communicate about what is happening depending on which aspects of the frame we wish to emphasize:

The woman is baking a cake (with the frame elements: baker + baked food
She is stirring in the pot (with the frame elements: baker + utensils)
She’s in a kitchen (with the frame elements: baker + location
…

Frames can also be more complex. For instance, they can propose a particular viewpoint on an event, allowing us to see the same event from different viewpoints. A classic example is the "buyer" versus "seller" frames, in which the same transaction can be viewed from the perspective of the buyer (e.g. "She bought cookies from my niece") or the seller (e.g. "My niece sold her some cookies"). Frames can also “evoke” a wide range of associations. For instance, sentences such as "COVID-19 is an invisible enemy" evokes frames about fights and wartime, and invites the addressee to make sense of the COVID-19 pandemic in such terms. The way a society perceives a particular issue may have important effects on policy making: if COVID-19 is seen as an enemy of the people, citizens will perhaps demand more far-reaching action from their governments. If however COVID-19 is seen as “a hoax”, people might resist any new policy decision.

In other words, if we want artificial systems to understand how humans perceive and make sense of complex issues, we need to identify how they “frame” particular events. Since language is one way to peek into the human mind, we can use people’s linguistic behaviors as evidence for how they frame reality. We are therefore working on a “Frame Extractor” that aims at identifying which frames people express in natural language texts written in Dutch, English, French, German, Italian and Spanish. Since these European languages are sufficiently similar to each other, we are working on a single repository of frames that can be shared by each language model.

One example is the Causation-frame, which (as you can guess from its name) frames an event in terms of a Cause-and-Effect relation. For example, in the sentence “Respondents believe that the coronavirus will cause an increase in income inequality in their country”, the Causation frame imposes a causal relation between “the coronavirus” (cause) and “an increase in income inequality in their country” (effect). The same sentence also expresses other frames: the Belief-frame (with “respondents” as the Believer, and the whole subclause as the Belief), and Future-frame (with the auxiliary will marking that the Causation-frame is a future possibility).

So how does our Frame Extractor work? A typical workflow starts by preprocessing and preparing a document using neurostatistical language processing tools (such as SpaCy), which includes tasks such as part-of-speech tagging (e.g. recognizing whether a word is a noun or a verb), tokenization (dividing a document into sentences, and sentences into words), and dependency parsing (identifying syntactic relations between words). Using another software library we developed in the MUHAI project, the result of preprocessing is then automatically translated into a symbolic representation that is readable both by human experts as by our computational platform called Fluid Construction Grammar (FCG), which is an open source special-purpose programming language for implementing grammars based on the notion of a “construction”. A construction can be thought of as a mapping between a form or a (syntactic) pattern on the one hand, and a meaning on the other. These constructions are used for identifying which parts of a sentence can be associated to the frames in our shared repository, and which frame elements are expressed in the sentence.

As an illustration, let us translate the aforementioned coronavirus example in Italian, and see how the Italian Frame Extractor is able to identify the Causation frame:

“Gli interpellati ritengono che il coronavirus causerà un aumento della disuguaglianza di reddito nel loro paese.”

Frames are typically identified by lexical or idiomatic constructions, which map a word or phrase onto a frame definition. In this example, we have a very clear lexical unit that triggers the Causation-frame: “causerà” (“will cause”). This construction will then introduce the Causation-frame, but now we still need to identify which Frame Elements are expressed. This task is performed by grammatical constructions: since the verb occurs in the Active Voice, we can infer from its lexical definition that its subject (“il coronavirus”) is the Cause, and that its Direct Object (“un aumento della disuguaglianza di reddito nel loro paese”) is the Effect. As can be seen in Figure 2, the Italian Frame Extractor indeed successfully identifies the Causation-frame and these two Frame Elements, and highlights the corresponding phrases in the text.

Depending on the kind of human-centric AI system that we want to develop, there are two ways of extracting frames that we call “phrase-based” and “meaning-based”. A phrase-based frame extractor can be seen as some kind of text annotator: given a document, it needs to identify which parts of the text may evoke a frame (“Frame-Evoking Elements” or FEEs), and then annotate which phrases are assigned to which frame elements, as we showed in Figure 2. While such a Frame Extractor is relatively shallow in the sense that it does not try to comprehend a text, it is already very useful for important applications, such as extractive search (e.g. for journalists or policy makers who need to browse through large amounts of documents). Meaning-based frame extraction, on the other hand, is not about annotating a text but translating it into semantic representations that are useful for other tasks that require comprehension. Again, many applications can be envisaged for this type of frame extractor: it can be used for automatically populating ontologies and event-based knowledge graphs based on textual data, for improving difficult tasks that require more semantic information such as Entity Linking and Reference Tracking, and so on.

Since the MUHAI project is all about human-centric AI, we will make our Frame Extractors publicly available to the research community as an open source software library, with a first release for English and Italian in March 2022; followed by a yearly update and release for French (2023) and for Spanish and German (2024).

Credits
Intro photo by Visual Stories || Micheile on Unsplash
Photo of Figure 1 by Jason Briscoe on Unsplash

MUHAI Visual Identity

2021-03-31T10:14:33+00:00

Paola Fortuna, Studio +fortuna

If the term “scientific” usually attracts our attention, the term “artificial” often alerts us, making us think of a world where human beings are at the margin and robots at the center.

In MUHAI we try to reverse this approach, starting by interacting with the term “artificial” in a different way. We do this visually, on the basis of a key concept of the project: “Breaking through the barrier of meaning” (Luc Steels, 2020). This vision goes straight to the part of our brain that thinks in images, by receiving images and returning images. In order to activate it we must explore what is underneath the surface of things and try and catch the idea we are trying to communicate, in its primary essence, Images reach the brain before words do.

Keys are needed to activate this area. We attempt to isolate some meaningful words in the in the texts produced for this project. Some keywords are therefore identified: human-centric / meaning / understanding / AI / language / research / process / dynamic memory / storytelling / creativity / relationship / abstraction / trustworthiness.
These concepts take us away from the cliché of a typical dystopian sci-fi picture of machines dominating human beings. Rather, they lead us to imagine an artificial intelligence that can help us improve our personal and social lives.

“Artificial Intelligence” on the internet, in its visual form, is blue, glacial. To “break through the barrier of meaning” we need to reduce the distance, by warming everything up with colors that welcome us and make us feel comfortable. We have to explore other colors, maybe warm and welcoming colors. We have to break through the barriers between colors, play with shades, so to make a color turn into another one seamlessly.

Once the colors have changed, the borders between them are erased, we then look at the words that make up the acronym MUHAI: Meaning / Understanding / Human-centric / Artificial / Intelligence.

MUHAI: Five letters, with an H at the center. The Human element is at the center of the project and becomes the center of its visualization as well. Just like a human being, the H begins to move on its own, upsets the balance of the word, and pulsates as a heart, transmitting passion and empathy. It also blinks like a watching eye, that looks at Artificial Intelligence in a new light.

To keep this pulsation imprinted even in a static subject, we pick a fragment of the movement that transmits the idea of flowing. And we remove the last hints of rigidity by rounding the edges of the
typeface.

To make this happen, we had to move away from the surface of things, in the urgency of conveying a new AI, where the Human is central also in its visualization.

Visions of the Future

2021-01-19T16:40:55+00:00

It started in the summer of 2030. Cecile’s daughter Lucilla had visited her mother’s wonderful home in a small town in the south of France and noticed that she was occasionally very absent-minded, forgetting things and behaving in a disoriented and erratic way, even for routine every day household tasks like cooking. Lucilla found a stove turned on without anything on it, several cups of half-drunk coffee around the living room, a pot with raw eggs and sugar. Her brother Juha had noticed similar unusual situations, including a chicken in the oven that had been roasted until black.

Cecile was only in her late sixties and in very good health, except for a weak shoulder which made some tasks harder, such as picking up a heavy pot. Apparently the loss of attention and memory occurred only during short intervals, a few times a week at most, but it was an ominous sign that her brain was aging too rapidly. Lucilla and Juha knew that their mother would ferociously resist leaving her house for a senior home, particularly because she enjoyed cooking elaborate dinners for friends and family in the great French cooking tradition, based on her own extensive repertoire of recipes with vegetables from her own garden. The past decade she had adopted a vegan orientation and had become known among vegan cooking aficionados for her novel recipes.

After consulting her brothers and sisters, Lucilla solicited the help of Ursula, her technically inclined sister-in-law. During one of their earlier family dinners, Ursula had been talking enthusiastically about robotic cooking aids that had come on the market around 2022. They do not consist of a two-legged robot, which only exist in science fiction anyway, but a system of cameras and tools on robot arms that hover over the kitchen countertop and sink. The robotic tools could pick up objects, fetch ingredients in kitchen cabinets, cut vegetables, turn on and off the oven, mix dough, and do many other subtasks to help prepare a meal. They could even order ingredients in the style of Google’s Alexa, launched a decade earlier.

In the autumn, they visited a company, discovered by Ursula, that had built a human-centric AI system on top of existing robotic cooking aids. The system was called ’La Jefa’. It had a personal dynamic memory that it built up by interacting with its users in a natural way, by observing what was going on, engaging in dialog, and collaborating on cooking tasks. La Jefa supported a meaning-based dialog in French or other languages and came with a broad knowledge of cooking terminology, techniques, and basic recipes. Because La Jefa could understand the meaning of utterances and the context, it was able to deal with unknown words, unusual expressions, and grammatical errors, and was therefore highly flexible. Most important of all, it was able to understand what its users wanted, even if the instructions were vague or assumed a lot of common sense knowledge and knowledge of the contexts and intentions of its users. The language competence, the general common sense knowledge needed for carrying out recipes, and user-specific knowledge incrementally extended through natural dialog and situated learning.

Under the pretense of remodeling the kitchen, Ursula and Lucilla installed the robotic cooking tools as well as the La Jefa AI system while Cecile stayed for Christmas with Juha. They then started to experiment by inputting some of Cecile’s recipes and trying out whether La Jefa was usable and indeed extensible. After a week they were convinced that the system was working and was indeed incredibly helpful. They were optimistic that it would be a blessing for their mother. At the annual family gathering on the first of January 2034, the brothers and sisters proudly unveiled the new kitchen aid as a New Year’s gift to Cecile. She was at first taken aback and worried whether she would be able to use all this. But with the help of her daughters she started to practice step by step and she was gradually able to get the kind of assistance that she needed to keep enjoying cooking again. A critical feature of the assistant was that he could explain why he was doing things in a certain way, often reminding Cecile of things she had told him before. Her grandchildren enjoyed La Jefa as well and they enthusiastically did all sorts of cooking experiments while staying at their grandmother’s house over the summer holiday.

A year later, there was another New Year’s party at Cecile’s home and the whole family sat down for the traditional dinner. The star of the evening was La Jefa. La Jefa had picked up on Cecile’s habits and ways of talking and had learned the peculiarities of how she wanted to cook in the vegan style she was innovatively exploring. La Jefa had become Cecile’s trusted partner helping with physical actions that were becoming more difficult for Cecile, but also helping her to keep focused on the task, reminding her of the next steps, telling her where some of the utensils or ingredients were located in the kitchen, ordering all the necessary ingredients for a recipe, and much more. Meanwhile Cecile firmly remained in charge and directed the whole operation. It was her cooking. La Jefa would also prevent potential catastrophes, like forgetting to turn off the oven, and it could warn Cecile’s children if abnormal things were happening in the eating habits of their mother - a first sign that there might be something wrong. It was also wonderful that Cecile blossomed intellectually with the new challenges that La Jefa brought with it. Her absent-mindedness had diminished and she proudly showed off La Jefa to her friends, even teaching vegan cooking lessons to others in town.