Robert Porzel, University of Bremen.

One major motivation of explainable artificial intelligence (XAI) is the desire to make the predictions of black-box machine learning (ML) models more transparent. Adadi and Berrada conducted a survey of the XAI literature and created an overview of common XAI methods - all of the methods presented range over the internal mechanisms of a ML model [Adadi and Berrada, 2018], which can be called introspective explanations. More recently, there has been an increasing need to explain AI behavior to non-expert users of AI systems. Therefore, XAI needs to focus more on end-users as the recipients of the explanations and on how XAI methods can be evaluated from a human-centered point of view. This sheds a different light on estimating the quality of an explanation and how it can be tailored adequately to the given user. This view is supported by the argument that explanation should be understood as interactive conversations. The usefulness of explanations can depend on the context in which they are given, including the situation and the user. In domains such as everyday activities, human users interact with AI agents repeatedly over time. When the two disagree, helpful and succinct explanations of the AI’s behavior should be tailored to the user based on its memory of their previous interactions.

To foster a human-centered approach to XAI, individuals and their preferences should be considered by appropriate user models. These can encompass, e.g., a level of education, cultural background, familiarity and interest in technology, perception of the agent, mood, and social settings. To accommodate these factors, personalization of AI explanations might be crucial. This is particularly relevant in domains where users interact with the same artificial agent over a long period, such as smart home devices, robots in household and care domains, and other personal AI applications. If users ask for explanations because they expected different AI behaviour, there must be some disagreement between users and AI in their respective beliefs and knowledge, which is what we call their uncommon ground. Explanations should allow the user to realize where the uncommon ground is. In error cases, this can empower users to correct the AI’s behaviour for future interactions; in other situations, it can enable them to see that they are missing important knowledge themselves. To facilitate this, explanations by the AI should reflect its experience from previous interactions with the same user. Personalization can improve user satisfaction by using user models created with direct or indirect user input, i.e., user entry or automated systems that adapt to user behaviour. Although direct user inputs can have advantages, an agent that is not properly configured may be less acceptable, thus personalization through indirect adaptation might be preferential. We argue that it is thus necessary to consider how to generate extrospective explanations: explanations that are given with respect not only to the system itself but also take into account what the system knows about the user’s expectations and draw on experience from earlier interactions. This extrospective perspective on XAI is at the heart of our work, as we focus on domains where users interact with smart software, intelligent devices, and autonomous agents/robots repeatedly over time, e.g., smart home device and household. This, in a sense, entails that users have their ‘own’ personalized AI. Explanations in the context of repeated interactions have not been studied as much. Therefore, we seek to examine this challenging problem systematically.  There is a growing awareness in the XAI community that explanations ought to be considered and evaluated in the context of the user for whom they are intended. Previous works have pointed out the problem of implied-but-not-stated contrastive/counterfactual cases when humans ask for explanations as well as that there are many situations in which the user’s goal in asking for an explanation relies on understanding why the AI prediction differed from their expectation.

We claim that in the domain of AI in the home (e.g., smart home devices, household robots), extrospective explanations can provide more helpful information for end-users in a more human-centered way, enabling them to understand the reason for unexpected agent behaviour. We aim to achieve this by focusing on explanations for transparent, reasoning-based AI systems and explicitly modelling user and discourse context in the agent’s knowledge. If the AI predicts the same out- come based on the same situational knowledge and ontology in two equivalent situations, the respective explanations could differ based on the user asking for them: For each user, the AI would output only a part of their reasoning. Specifically, the AI should output those facts or inferences that it expects are most likely to surprise the user. This should be whatever fact or logical inference the AI has the least reason to believe that the user agrees with or is aware of. In other domains, however, there might be different reasons why explanations should be purely introspective. There are many areas of life where AI is or will be used, in which it will be important that explanations are consistent across different users or that they are a complete representation of the AI reasoning (e.g., medical or legal problems). Therefore, it will be essential to distinguish between explanations aimed at helping individual users understand local, surprising AI predictions and explanations aimed at increasing ML systems’ transparency.

[Adadi and Berrada, 2018] Amina Adadi and Mohammed Berrada. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6:52138–52160, 2018.

Full Paper in the papers page:

Laura Spillner, Nima Zargham, Mihai Pomarlan, Robert Porzel and Rainer Malaka. Finding Uncommon Ground: A Human-Centered Model for Extrospective Explanations. In Proceedings of the 2023 IJCAI workshop on Explainable Artificial Intelligence (XAI). 2023. BibTeX, pdf 

Interview with Katrien Beuls, UNamur.

We are pleased to announce that MUHAI researcher Katrien Beuls is now Lecturer in Artificial Intelligence at the University of Namur (Belgium) and we are happy to welcome UNamur in the MUHAI consortium! Read her interview for the University of Namur's magazine "Omalius".

Katrien Beuls is undoubtedly a fine example of the growing number of women in STEM (science, technology, engineering, and mathematics) careers. After a rather literary career, guided by her curiosity, she began studying computer science and became interested in computational methods for processing human language with the help of Artificial Intelligence (AI).

After an atypical curriculum, her meeting with Luc Steels (VUB), a pioneer of artificial intelligence in Europe and an expert in evolutionary computational linguistics, and an international experience as a student and post-doctoral researcher, Katrien Beuls has just moved to UNamur.

"A position as 'Lecturer in Artificial Intelligence' was vacant. I applied and here I am! I found it interesting that the job description did not restrict the function to a specific sub-field of AI. I find UNamur extraordinary, because of the human size of the institution, which allows for a lot of interaction quickly, as you get to know everyone quickly! I also had the impression that there was a certain open-mindedness in the Faculty of Computer Science and that they were interested in new methodologies. I really feel that UNamur is an excellent place to create my own interdisciplinary research team.

A passionate researcher
"I am trying to build intelligent systems, so-called cognitive agents that implement the 'perceive', 'reason', 'act' cycle that constitutes the decision process.  These intelligent agents have to solve communication tasks in teams.  The aim is to build a common language to refer to a situation, objects, etc. This is done in the form of a game according to an established protocol.  An agent chooses a topic of conversation and has to ask himself the question: what do I have to say so that the other knows what I am talking about?  They have perceptual abilities but see the world or segments of it without labels and have no language system.

This interview was conducted for the "Expert" section of the University of Namur's Omalius magazine #27 (December 2022).

Photo of Katrien Beuls - Credits: © Christophe Danaux

© UNamur

Anna Morbiato, VIU.

Imagine you're at a house party. You've just met someone new and you want to make a good impression. So, you ask them "How's it going?". To a machine, this phrase might seem like a simple question, but to a human it has a whole range of meanings. It could be a polite greeting, a genuine inquiry about how the person is doing, or even a subtle way of asking them to leave. The machine might not be able to pick up on the nuances of the conversation, but a human would be able to decipher the true meaning behind the words. That's pragmatics in action!

Pragmatics is the study of how we use language in context. It looks at how the meaning of words and expressions may change depending on the situation and the people involved. It also considers how our intentions and our social relationships affect the way we communicate. It is like a party trick - it's the art of knowing how to communicate the right thing at the right time!

Pragmatics is the focus of the paper Pragmatics of Narration with Language, which is part of the MUHAI white paper Foundations for Meaning and Understanding in Human-centric AI. Specifically, the paper looks at how research in pragmatics and cognition over the past 50 years has helped us to better understand natural languages, narratives, and language processing. The paper explores how factors such as the speaker's communicative aims, perceptual experiences, stance/viewpoint, as well as cultural and social aspects of communication, can affect how humans use language.

One of the aspects the paper looks at is coreference disambiguation, which is essential for parsers and machine learning when it comes to understanding natural language. Imagine you're making pizza with a friend; she is kneading the dough and you want to check on it. So, you ask "How's it going?". To a machine, this phrase might seem like a simple question, but if the parser incorrectly disambiguates the pronoun "it" as referring to the friend, instead of the dish, it could lead to confusion. The machine might interpret the question as asking about the friend's wellbeing, instead of the progress of the dish. Coreference disambiguation is the process of determining which words or phrases in a sentence refer to the same entity. This helps the parser to accurately interpret the sentence and understand the intended meaning.

Another both interesting and challenging realm is that of speech acts. Speech acts are the ways in which we use language to express our intentions and convey subtle or implicit meaning, such as orders, suggestions, promises, and requests, as well as a variety of other communicative goals. Getting back to your cooking session with your friend, imagine your pizza is baking in the oven, and your friend wants to check on it. So, she suggests: "Maybe it's time to take it out now?". To a machine, this phrase might seem like a simple suggestion, but you would instead be able to detect whether it is a nice way of hinting that the pizza is burning (and you should have already turned the oven off). Pragmatics is like a chef's special ingredient - it's the art of knowing how to season a dish just right so it tastes delicious!

That's why pragmatics is so important, both for humans and for parsers: because it helps to make sense of the context of conversations and interactions. The goal is the development of deep learning models that can understand natural language also taking into account these nuances of human interaction. This technology is still in its infancy, but it shows great promise for improving the accuracy of machine learning models in interpreting natural language.

Credits
Intro image - photo by Andrea Piacquadio via Pexels

Alexane Jouglar, University of Namur.

Source: https://dictionary.cambridge.org/dictionary/english/cook

Cooking is an act that we perform in our everyday lives to produce (delicious) dishes ready to eat. Many machines exist to help humans cook: slow cookers, multi-cookers, pressure cookers… All these machines are really helpful, but they need the direct intervention of a human. Currently, It is not possible to give a recipe to a machine, place the machine in the kitchen, and ask it to cook the dish. The reason for this is that recipes include a lot of background and implicit knowledge that humans gradually acquire by practicing and interacting with the world and that cannot be understood from the only lecture of the recipe. Let’s take an example.

Source: https://chocolatecoveredkatie.com/vegan-chocolate-chip-cookies-recipe/

As we can see in the figure above, the recipe instructions start with “Combine all dry ingredients in a bowl”. As humans, we know what dry ingredients are. We also know that by “all”, the writer means “from all the ingredients that are listed above”. For a machine, this is a lot harder. It needs to be equipped with a lot of background knowledge. That's precisely the challenge posed by a groundbreaking new benchmark introduced in a recent paper published at LREC-COLING 2024:

Nevens, J., De Haes, R., Ringe, R., Pomarlan, M., Porzel, R., Beuls, K., & Van Eecke, P. (Accepted/In press). A Benchmark for Recipe Understanding in Artificial Agents. In The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation

This benchmark is the result of a collaboration between the Vrije Universiteit Brussels (VUB), and the University of Bremen (UBremen) of the University of Namur (UNamur). It aims to evaluate whether artificial agents can understand and perform everyday cooking activities. It includes several components:

• Corpus of Recipes: A collection of 30 recipes of varying complexity.
• Procedural Semantic Representation Language: This language helps formalize 38 cooking actions in a way that machines can understand. It breaks down recipes into precise steps and provides a framework for representing cooking processes.
• Kitchen Simulators: Both qualitative and quantitative simulators recreate kitchen environments where agents can practice cooking. These simulators simulate the physical aspects of cooking, such as ingredient interactions and cooking times.
• Evaluation Procedure: A standardized method for evaluating agent performance.

The main task of the benchmark is to translate natural language recipes into a series of cooking actions that can be executed in the simulated kitchen to produce the desired dish. This translation process requires the agent to reason over multiple factors, including the recipe text, the state of the simulated kitchen, common-sense knowledge, and domain-specific cooking knowledge.

Success in this benchmark requires a combination of natural language processing and situated reasoning. By mastering the art of cooking, artificial agents can become more versatile and helpful in various real-world scenarios.

Source: Canva

The introduction of this benchmark represents a significant step forward in the field of artificial intelligence. It challenges researchers to develop systems that can understand natural language and apply that understanding to practical tasks like cooking. As these systems improve, they have the potential to revolutionize various aspects of our lives, from personal assistance in the kitchen to industrial food production.

Discover the full paper in our dedicated section: Jens Nevens, Robin de Haes, Rachel Ringe, Mihai Pomarlan, Robert Porzel, Katrien Beuls and Paul van Eecke. A Benchmark for Recipe Understanding in Artificial Agents. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti and Nianwen Xue (eds.). Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024, 22–42

Anna Morbiato, Venice International University.

In the dazzling realm where cutting-edge technology meets the world of culinary arts, precision is not just a preference – it's the secret sauce to success. Imagine a futuristic kitchen where robotic chefs whip up gourmet delights with a mere tap on a touchscreen. Now, in this technologically driven culinary landscape, the unsung hero emerges: accurate tracking of coreference. Like a culinary GPS, it navigates all complex recipe transformations undergone by ingredients, ensuring that every ‘it’ and ‘they’ (as in ‘put it in the fridge’ or ‘they are ready when a fork slides in easily’) points to the right flavour-packed entity. Let's explore why this seemingly subtle linguistic skill is the backbone of a seamless, tech-infused gastronomic experience.
Take a simple Italian pasta recipe step: 'Boil a pot of water, add some salt, then add your favourite pasta type. Once it's cooked, drain it and then add it directly to the tomato sauce.' While us readers have no doubt what 'it' refers to, it is not as straightforward for a machine to understand that 'it' refers to the pasta and not to other nouns like 'pot,' 'water,' or 'salt.' Still, it is crucial that the right item is added to the sauce for a successful execution of the recipe.

Accurate tracking of coreference becomes crucial in a technologically driven culinary landscape. But this comprehension comes with its set of challenges. Recipes are full of pronouns and other ways to refer to items, ingredients, and tools. However, these are sometimes far from transparent and often include not only pronouns, but also even zero anaphors (Ø) (namely using no words at all!). The same sentence as above could work also with no 'it': Boil a pot of water, put some salt, then add your favourite pasta type. Once Ø cooked, drain Ø and then add Ø directly to the tomato sauce.’ Not only ingredients, but also intermediate products of each step (resultant objects, like a mixture or a cream) are frequently left unmentioned, resulting in potential ambiguity. This phenomenon is not marginal; in fact, zero anaphors (also referred to as zero pronouns, null elements, or implicit entities) are more frequent in recipes as compared to other genres. This is especially evident in pro-drop languages like Italian and topic-drop languages like Chinese, where zero anaphora is widespread and often requires inference for correct interpretation. Further challenges include partial coreference phenomena (also known as bridging) as well as evolutive anaphors, namely linguistic expressions that refer to an entity while also indicating a change in its properties or attributes over time. Evolutive anaphors are very common in culinary narratives, where they typically refer to an ingredient or component of the recipe which, in the meantime, has undergone some transformation.
Imagine we want to make baked potatoes. The recipe starts by telling us ‘Wask and peel the potatoes.’ Then it goes on with the cooking procedure, which includes garlic, warm milk and butter. The last passage says: “Blend with a potato masher or electric mixer until potatoes are smooth and creamy.” For sure, the entity denoted by the term potatoes in the ingredients and that denoted by the same word in the last passage are very different. Then why do we still use the word potatoes? And why does it sound odd if we do the same in a sentence like ‘Juice the apples, then put them onto a pan.’?

Unravelling the complexities of anaphoric references in recipes is the objective of the paper ‘Pragmatics as the secret ingredient for NLP: a cross-linguistic study of reference tracking and evolutive anaphors’. The paper provides an in-depth exploration of coreference encoding and resolution in the domains of linguistics and Natural Language Processing, with a particular emphasis on zero pronouns, partial coreference phenomena, and evolutive anaphors. Additionally, it presents an analysis of the flow of information in recipe texts, as well as of the type, frequency, and nature of anaphoric devices used in authentic recipes drawn from food blogs. It considers texts in English, Italian, and Chinese, which significantly vary in terms of coreference tracking mechanisms and demonstrate different levels of dependence on zero anaphors and inferential processes. The paper gives particular attention to evolutive anaphors and the role played by inference and world knowledge in coreference disambiguation. Recall the pasta recipes above: the pronoun it involves a change in properties of the pasta, from being hard to being soft after cooking. Crucially, inference plays a significant role in coreference disambiguation, in addition to, or in place of, lexical and grammatical encoding. In fact, it is world knowledge and common sense that enable the reader to understand that the anaphors refer to pasta, and not to pot, water, or salt. Any AI systems aiming to effectively address anaphora resolution and achieve complete textual comprehension must integrate inferential processes at a certain stage.

Robert Porzel, University Bremen.

Trust is an important factor in human-centric artificial intelligence – especially for the success and effectiveness of a collaborative task in which the participants rely on each other to achieve specific sub-goals. For example, in household environments, such as a kitchen, mistakes can be made by either party that could not only lead to failure to complete the task, but even to injury through various hot or sharp appliances. Trust in a new system or technology is critical to its success, since people tend to employ systems that they trust, and reject systems that they do not trust.

In the last few years, artificial agents, such as vacuuming robots, have become more common in household environments, and assistants for more complex tasks as cooking or cleaning are being developed. To ensure that these new systems will be accepted, it is important to explore how much people trust an autonomous system to handle these tasks, how this trust changes during use and what factors lead to an increase or decrease in trust. Toward that goal, it is important to find applicable measures for trust. For this, we propose measuring a user's trust in an artificial collaborator during cooperative cooking tasks by analysing the tasks delegated to the artificial partner during collaborative execution of a recipe.

As the delegation of tasks among humans relies on trust, we propose that the tasks given to the artificial collaborator, e.g. while preparing a meal, can supply information on the level of trust the human has in them. If the human assigns intricate, dangerous or important tasks to the artificial agent, e.g. heating or cutting an ingredient, this would indicate, that they trust this partner to complete the task successfully. Should they only delegate minor tasks to the robot – for example, wiping the counter – it indicates, that the robotic partner is only trusted to fulfil simple tasks where errors could easily mitigated. Toward the goal of measuring trust based on task delegation, three different aspects of a task that could influence a human’s tendency to delegate it were chosen in our approach: difficulty, risk and possibility for error mitigation. In addition, it was deemed relevant if a human would supervise the artificial collaborator during a task or even intervene. In addition discount factors were considered that might convince a human to assign a task to a robot even though they do not completely trust the robot, e.g. tediousness of a task or inability to complete a task themselves. These aspects were combined into a basis for a scale, that can be used to determine the level of trust the human put into the artificial partner when delegating this specific task to them.

To observe humans during cooperative cooking with an artificial partner, a VR application was developed in the Unity game engine for use with an Oculus Quest HMD. In this application the user is placed in a kitchen environment together with a virtual robot. The user can interact with various objects in the kitchen by grabbing them with either their hands or the controllers and then complete various cooking tasks by moving them in appropriate ways -- e.g., moving a whisk in circular motions through a bowl containing the different ingredients to be mixed. In addition, the user can order the robot to fulfil any of the needed cooking tasks for recipe completion -- e.g., portioning a certain amount of an ingredient into a bowl -- or some supporting such as cleaning, tidying or fetching objects for the user. For these orders a delegation-type interface is used, where the user orders the robot to fulfil a task in a declarative manner, but is not required to give details on how the task should be completed.

In the future this metric could be part of a bigger set of measures for trust specific to cooperative tasks, that includes other aspects such as the phrasing of orders given to the robot. Similarly, it could be modified for further household tasks, that could in the future be assigned to household robots. Predictions made by a graphical model based on these metrics could also be used to adjust robot behavior at runtime to calibrate trust to the appropriate level for optimal cooperation. The described test environment and scale could be used in the future to explore different robot appearances and behaviors and how they affect trust, as well as trust development over time when the human can observe the robot complete tasks successfully or make errors.

Mihai Pomarlan, UHB.

A recurring theme of AI research has turned out to be that what should be easy often is not. Consider this question – "what can I cut a stick of butter with?" You probably already thought of an answer, and if pressed, you could invent more creative ones. A string might do, or the edge of a glass perhaps if no knife is available – though you might protest if I suggested a jar's edge. It will be annoying to get the butter rests out of the threading.

You can answer such questions as the one above, and even employ some creativity in the answers, because, presumably, you have built a sophisticated model of the world and the things in it through your experience of interaction with them. You have an idea of what an object can do, and how it will change in various situations. Now that you have this model, it is second nature to reason with it. Could we implement something similar inside a robot to help us with housework? As always, let's start with moderate ambition. Could we implement a model of "normal" object use, so that our housework robot could at least give the (to us) obvious answer? We'll worry about whether a robot should be creative, and if so how, some other time.

But if all we want is a database of normal object uses, surely we have such a thing already. You could google "what can I cut a stick of butter with?" and get good answers. If google can answer it, surely a robot also can. And yet ... "what should be easy often is not". It's up to you to select which of the answers are relevant to you, so finding a result with google is often more of a collaboration than it first appears – and both partners should have some idea of the thing searched for. When I ran this experiment, the first article google found was about "cutting in butter", a technique to mix in butter with the dry ingredients for baking. This is not quite what I meant, so I can go down the list of search results to find one that tells me what I expected to find. Would a robot who did not already know how butter and cutting work be able to use these results? And what google finds is in natural language, rather than a nice, well-structured, immediately usable message for a computer program.

So let's say google is a bit too tricky for our robot, but surely there are large databases out there which cover this sort of everyday commonsense knowledge. Indeed, since commonsense reasoning is recognized as a challenge in AI, there are many efforts towards building commonsense knowledge resources – and this time, the data is represented in machine readable format. Querying and interpreting the results would be easy for a robot to do. The problem is that, were the robot to ask our cutting butter question, he's likely to not get any answer. It turns out, existing commonsense knowledge resources are mostly useful to assisst search engines towards organizing the websites they crawl through into a more semantically rich web. For example, a commonsense knowledge resource might know of many famous people, and that they were human beings, and that human beings have birth dates. Or it would know of countries, and their capital cities. Such information is useful to construct a brief summary of knowledge about an entity, and will also assist in retrieving more documents about that entity – but now we are again in the area of a search engine providing results for a human being to select from. And the human being asked for the search. "What should be easy often is not" – in this case, because if it is easy for a human there is no point in firing up a search engine for an answer. As a result, what should be a commonsense knowledge resource ends up being a resource for trivial questions instead.

This does not make existing commonsense knowledge resources useless to our robot, but we need to add some more knowledge to them first. Some of this object usability knowledge was obtained by colleagues working on a related project, which involved the development of games through which human players could reveal their preferred object combinations when performing various tasks. Some valuable information we obtained from linguistic resources that describe the roles objects can play in various kinds of events, and restrictions on what objects can play which roles. Finally, we used a cognitively motivated ontological foundation to construct our model upon, which now allows the robot to reason with it and answer questions such as "what can this object be used for?", "can this object be used for a particular task?", and "what can this object be used with when performing a task?''. Altogether, these are the existing knowledge resources we have used so far are: the CommonSense Knowledge Graph (CSKG), the linguistic resources VerbNET, WordNET, the data from the game with a purpose ToolFeud that our colleagues collected, the DOLCE Ultra Lite foundational ontology, and the SOcio-physical Model of Activity (SOMA) that we are developping in a related project. After also some manual input and corrections of data collected semi-automatically from CSKG, we obtained a new knowledge resource, SOMA_DFL.

There is always more work to be done, of course. Part of that work, ongoing at the moment, is to incorporate some causal knowledge in the SOMA_DFL knowledge base: what happens to an object if some action is performed upon it, how does the quality of an object affect the event it participates in? None of this refers to any advanced knowledge of science; what we want to have in SOMA_DFL is the kind of knowledge that is so useful, and so obvious – to human beings – that almost no one thinks worth writing down. What should be easy often is not – a robot trying to do houeswork will need that knowledge, even as it lacks a natural intuition for it.

In any case, SOMA_DFL is available at the repository https://github.com/ease-crc/ease_lexical_resources. If your robots want to know how to choose tools for some job, give it a try. If they don't find an answer, let us know!

Credits
Intro image - Photo by Stoica Ionela via Unsplash

Robert Porzel, UHB.

If the proof of the pudding is in the eating then the ultimate test for understanding an instruction is its proper execution. This view greatly expands the scope of natural language understanding beyond the usual syntactic and semantic analysis. In this part of the MUHAI project we seek to operationalize the basic principles of human-centric AI so that machines will be able to understand how to perform everyday actions in the cooking domain. This involves moving away from executing fully explicit standardised instructions towards understanding instructions conveyed through natural language dialogues. The key challenge here is the integration of world knowledge and pragmatic inferencing into the understanding process, both on the level of language processing and on the level of task execution. For example, the knowledge that chopping a cucumber involves the use of a cutting board and a knife, and presupposes a specific orientation of the cucumber, as well as a conventional slice thickness, is not explicitly mentioned in a recipe, but is essential to carrying out the task and must therefore be inferred from common sense knowledge. Also the build-up of knowledge that generalises across recipes and ingredients is of importance, as it is a precondition for adapting existing recipes to given constraints, and ultimately for the creative design of novel recipes.

In order to achieve these goals we will define two kinds of benchmarks:

1. one that consists in mapping between existing recipes formulated in natural language and actions executed in the VR world
2. one that allows us to evaluate a new recipe design or variant proposal

As in all parts of the MUHAI project the notion of meaning-based and human-centric narratives also applied in the cooking domain. These narratives give meaning to collections of experiences of a virtual agent, i.e. object perceptions, body postures, force dynamics, visual processing and structured data collection, i.e. recipes, images and procedures. Building narratives requires the integration of multimodal sources of input (text, image, sound) and pattern detection in a model of constructional language processing. Constructions will be used as the basic representational unit in which all of these sources are combined. The outcome of constructional language processing is a semantic analysis, including identification of goals, plans, actions, objects, time and causation. The set of analyses make up the starting point for narratives in the domain that can be integrated with the personal dynamic memory in order to truly understand them, in the sense that they can be mapped to a series of low-level actions that can then be executed by a simulated agent in the VR kitchen environment.

To demonstrate the potential of this approach MUHAI will  develop two applications for recipe execution and design:

1. Recipe execution - This application consists in executing recipes expressed in natural language in a VR kitchen environment. This requires mapping between a recipe (i.e. a sequence of instructions) and a sequence of low-level actions to be executed. The application will involve constructional language processing, consultation with the personal dynamic memory for pragmatic inference, and planning the execution of the concrete cooking actions. The application will be evaluated on the benchmarks described above
2. Recipe design - This application is situated in the domain of professional recipe design. In the first part of the project MUHAI will focus on the challenge of building a virtual agent that can act as an assistant chef. This digital assistant needs to integrate technical cooking knowledge with a considerable prior memory of recipes, previously successful and unsuccessful variants, cooking procedures, and cultural context. Most importantly, it needs to do so in an explicable, transparent manner. In a second phase, the focus will shift to a more challenging task that embraces even more aspects of human-centric AI, namely that of recipe design. This is a capacity that goes beyond skill and knowledge and introduces creativity.

Paul Van Eecke (VUB)
Katrien Beuls (UNamur)
Tim Brys (VUB)
Adapted by Folco Soffietti (VIU)

Source: Pexels, modified by Folco Soffietti  

Adapted from the article “Studeren zonder ChatGPT, daarna verstandiger werken met AI” published on the Knack on the 9th of April 2024.

The Integration of AI into curricula cannot neglect a sound domain knowledge, good language skills and thorough knowledge of the scientific method.

In Belgium, the Rector of the University of Gent announced that from next academic year, students will be allowed to use generative AI in their thesis. This sparked a debate within the academic community, questioning the purpose and methods of university evaluation in the age of AI.

Generative AI is supposed to be able to reshape every aspect of the research cycle, but is it so? MUHAI’s researchers can help you dive a bit deeper in the matter.

Generative AI systems such as ChatGPT can produce texts that are sometimes indistinguishable from those written by humans, but still have difficulties in imitating logical thought processes. Even so, let us assume that generative AI can still replace important elements of the research process. The detection of AI use is becoming increasingly accurate, just like plagiarism. And what student will risk losing their degree due to fraud, possibly even years after the date?

Even so, we can still make the exercise of assuming that it is not feasible to ban the use of generative AI. Then two paths open up: allow AI use or discard the thesis. Contrary to the Rector of the University of Gent, some professors consider that understanding and processing of knowledge cannot be outsourced to ChatGPT when studying. The risk is, in fact, to no longer acquire these non-negotiable basic competencies. They are crucial in an academic education. Therefore, if the use of generative AI cannot be restricted in practice, it seems that the master's thesis shall be replaced.

Until recently, the master's thesis was the culmination of a long educational curriculum in which a set of skills and understandings were gradually built up in students. In general, we do teach skills for which there exist digital tools that can perform them better or more efficiently. For example, we still learn spelling in primary school, even though spell-checking programmes exist. Why? Because spell check only makes sense if you can already spell yourself and write an intelligible text. The same with arithmetic: we still learn arithmetic rules because a calculator is only useful if you have the necessary mathematical understanding yourself. These tools can help us avoid mistakes and make our work more efficient. But if we cannot interpret the output of these tools, they are completely useless.

It is no different with generative AI at the master's level. In a master's programme, a student is expected to grasp professional literature, acquire research skills, develop critical thinking, and so on. Only someone who has already acquired these skills to a considerable extent can use generative AI responsibly at the level expected of a master. How else can that person interpret what the AI system produces? An incompetent writer cannot judge the quality of the generated text, let alone improve it.

if we want our students to be able to use generative AI competently and responsibly in the workplace, they will still have to acquire the basic equipment of the master's degree. Shortcuts and distractions of generative AI will have to be ignored for a while with the higher goal in mind: being formed into a competent expert in the chosen field of study.

If the master's thesis won’t be a reliable way to test academic growth, there are fortunately alternatives. Evaluating throughout the year, or final exams. Either way, generative AI should not mean the end of the goal of shaping students broadly and scientifically. And this, regardless of a specific tool.

Carlo R. M. A. Santagiustina, Venice International University.

-Continues from Part 2

To answer the questions raised in the previous chapter of this article in a synthetic way, here follows a short-list of major implications of the aforementioned changes to data access policies. To grasp the societal impact of these changes, I selected two examples of social media data-based and community-centric projects. Today, with the current social media policies, the collection of user-generated data required by these projects would have been impossible to afford. Both projects are ongoing and are related to -and contributed by- MUHAI. The first one, called AquaGranda: A digital community memory, started in 2020 from the EU project Odycceus, after the extreme tide of 187cm that flooded Venice in November 2019, and progressively grew involving other EU projects and research institutions, such as MUHAI, and recently received an Honorary Mention at ArsElectronica, for the European Union's Citizen-Science Prize. The other one, called MUHAI Inequality Observatory, started in 2021, and is being developed within our consortium by a multidisciplinary team of researchers. Both projects focus on topics that are undoubtedly relevant to a wide audience of citizens and institutions, respectively, the impact of extreme weather events on social conflicts, and the understanding of the public perception of social inequalities and intersectionality.

Democracy wanes – access denied, our cause remains

For projects like the aforementioned ones, free access limitations to user-generated  data from social media platforms, such as Twitter, can produce detrimental effects that, besides damaging scientific research, can directly affect communities and their economies at multiple levels. By limiting the generation of externalities that can produce widespread societal benefits, such as:

• Transparency: Free access to social media data allows researchers working on the perception of topical issues, such as the impact of sea level rising on coastal communities or that of social inequality on life expectancy, to gain insights into the expressed views, expectations, preferences, reactions and feelings of the people being (actually or potentially) affected by phenomena that are shaping our collective present, and will likely determine our future, as well as that of other species. Social media data access is also crucial for understanding the dynamics that push some to express their opinions and others to remain silent on issues that may directly affect them, such as climate change and social justice. Moreover, acknowledging the relevance of a particular issue for a wide spectrum of the population can be an incentive (and pressure) for representatives and governance bodies to activate, taking into consideration people’s expressed needs and concerns in relation to these topical issues. More generally, denying or limiting social media data access to researchers can damage evidence-based policy-making, for example with respect to issues related to climate change and its effects on citizens’ daily life.
• Innovation: Open access to social media data fuels innovation in various sectors that build on top of academic and non-profit research, such as the AI industry. If researchers can’t access social media data they won't be able to develop new tools and models to analyze this data, which are widely employed, not only in academia. Furthermore, charging for data access could disproportionately damage and deter innovative research projects with unconventional or exploratory purposes related to social media and their data, which may not have guaranteed or clearly foreseeable outcomes or applications.
• Accountability of social media platforms and users: Social media data can be used to hold platforms accountable for their impact on society, including issues like mis- and dis-information, trust erosion, and algorithmic biases. Restricting access to researchers could make it harder to observe, assess and address these issues.
• Equal access and equal opportunities: Non-profit organizations and academic researchers with limited resources might struggle to afford social-media data access. This could further exacerbate knowledge and information inequalities, especially in poorer countries.
• Knowledge consistency: If different social media platforms adopt different access policies, data sources might become inconsistent, making it difficult for researchers to compare them, or to analyze their dynamics and evolution across time, by integrating their findings across multiple platforms.
• Civil society and community engagement: Access to social media data is crucial for NGOs, IGOs and other non-profit projects for understanding public discourse and socio-cultural movements. Without social media insight, civil society projects might struggle to engage citizens effectively in participatory deliberation or policy-making processes. Moreover, the ability to see how groups of citizens deliberate through social media is essential for facilitating understanding among groups. Lack of access to data could hinder efforts to bridge divides.
• Accountability of elected bodies and representatives: Social media conversations provide citizen’s feedback on policies and their perceived effects. It is therefore important for holding representatives, governance bodies and institutions accountable and reactive. For example, if data access is limited potential abuses may go more easily unnoticed.

As you may have understood, open and free access to social media data for academic, non-profit, and civil society projects is essential for maintaining transparency and ensuring accountability in democratic countries. For this reason we must protect it, and you should help us in doing so.

The opinions expressed in this article are those of the author. They do not reflect the opinions or views of anyone else.

Carlo R. M. A. Santagiustina, Venice International University.

-Continues from Part 1 

Having worked in the last decade on several EU research projects, like MUHAI, which  aim to study social phenomena, such as inequality perception, through user-generated data from social media, I feel the need to put you on alert. Social media, and the information they collect, are certainly formidable tools for studying and comprehending social phenomena, but, unfortunately, they are also formidable artifacts for attempting to influence and manipulate online crowds. Therefore, they should be considered critical (dual-use) technologies in relation to freedom of expression, but also in relation to social justice, peacekeeping and for the proper functioning of public institutions as well as of markets. This is especially relevant in liberal democracies that,  from this point of view, are particularly vulnerable on both sides.

Social Media platforms can be considered the contemporary and digital equivalent of the agoras in ancient Greece, and just as in ancient agoras some people discussed public affairs while others did private business, the same occurs nowadays in these platforms. Both usages are per-se legitimate. But differently from what occurred in the past, nowadays citizens, representatives,  institutions, and civil society have little, if no, control on how these private online agoras are employed, and on the scopes for which the data generated by citizens is sold to (and used by) third parties, for political, social or commercial exploitation. Social media companies are certainly required to comply with privacy and data protection regulation, which may partially protect users, as single individuals and organizations, but certainly this type of protection does not shield us as individuals belonging to, or identifying with groups or communities, based on life-styles, values, beliefs, preferences or any other form of shared identity or common behavior. Because, once user-data is aggregated and anonymized, it can be used for all sorts of purposes, from profiling, categorization and personalized advertising to political influencing campaigns targeting groups of users that are similar in terms of one or several aspects of their online behavior and identity. What is known about social media users at the aggregate level can hence be used to affect their views, preferences and behaviors. Since these platforms are populated by millions of citizens from dozens of countries, including totalitarian countries and façade democracies, many actors and organizations may have incentives to profit on top of their opacity, by using these platforms and their data for manipulating opinions, either of large and varied user populations, or, of carefully targeted influential audiences.

Virtual clashes roar, divergent views explore, data locked, voices sore.

Some social media platforms, like Twitter, have become for the aforementioned reasons virtual -but no less violent- battlegrounds. Where armies of users and bots driven by divergent interests and views, like pro-Russians and pro-Ukrainians, confront each other in harsh debates for setting the agenda of politicians and governance bodies or for influencing the opinions of consumers and citizens that use them. Other platforms, like Parler, stimulate by design the self-selection of aligned users and contents through homophily.  Shared preferences, expectations and views that are sufficiently widespread among their communities, shield community insiders and their partisan convictions from the risks and wonders of communicating with people with differing opinions. 

Unfortunately, in most cases, the growth of these private platforms and of the techniques for analyzing the rich and big data generated by their userbases, was followed by further restrictions for freely accessing the latter for non-profit research purposes. The recent diffusion and commercial success of some general-scope LLMs, which can also be trained using textual data from social media, has unfortunately exacerbated this process of user-data privatization and monetization. This is especially problematic for social science research and other non-profit projects, like AquaGranda: a Digital Community Memory, which, given their nature, do not (and cannot) aim to generate a profit from the usage of social-media data, and therefore have great difficulties in covering data access costs. This category is rather broad, and, besides researchers from academia and public research programs, it also includes researchers working for IGOs and NGOs, artists and activists, advocacy and volunteering groups and other non-profit organizations, who are also those whose activities and outcomes can possibly generate the greatest collective benefits and positive externalities for the public.

Short-circuiting non-profit research programs that study social, political, economic or socio-natural phenomena through the data generated by users on social media is easier than it may seem, and some platform managers and shareholders may want to do it simply to ask (data-) protection money also to academia, for continuing projects already underway.

Social media data policy changes are not simply due to privacy and data protection regulations and related concerns of the platforms that collect user-generated data. But rather, they often stem from the desire to further privatize and monetise the value of data contributed in exchange for no compensation by online communities, that is: your data. The data that you, your friends and your family, among others, generated by posting contents and commenting on each-others’ posts in the last two decades. Unfortunately, this process is not limited to Twitter, which recently withdrew free access to verified research projects, by deleting academic projects and their credentials from the Twitter developer portal without any notice. For example, also Reddit policies were recently changed, and this obliged the free PushShift archive to close its doors also to research projects. As strange as it may seem, Reddit moderators can still access the PushShift archive and its APIs but researchers from academia cannot. These are only two highly visible instances in a rapidly evolving social media landscape, which seems to want to make itself more and more profit-oriented and opaque.

At this point, a couple of concrete questions may be running through your mind:

• What is lost when social media platforms change their free data access policies to researchers and other non-profit organizations?
• What could occur if, because of the increasing data-access costs and constraints, civil society and citizen-science projects are denied the opportunity to see and comprehend how communities organize, debate and deliberate through social media?

(End of Chapter 2 – Find the answers in the next and last chapter)

Carlo R. M. A. Santagiustina, Venice International University.

From climate change to inequalities: Exploring the societal value of social-media for understanding public concerns and the perceptions of anthropogenic phenomena at a time when user data is increasingly monetized.

Web weaves lives, spheres interlace, online's intimate embrace reshapes human space.

In the last two decades, the Web has become a multilayered architecture for distributed communication that connects more than 5 billion human beings. Online communication is nowadays ubiquitous and permeates all spheres of our daily activities: from family relations to work activities; from hobbies to political propaganda; from volunteering to advocacy. Online communication is not simply answering a functional role related to people's insatiable need for information or gossip, it can trigger, permeate and shape intimate spheres of our life, producing profound and widespread transformations at the individual and collective level, shaping our identities, understandings, beliefs and preferences.

The diffusion of smartphones and of social media apps, has rendered communication increasingly multimodal and multimedial. Through our online posting activity, social, cultural, political and economic phenomena can get easily intertwined and percolate from physical to virtual spaces. For example, they can flow from news platforms to social media, to markets and back again to the physical world, affecting its abundances and famines.

On the Web, what might seem volatile as, for example, an opinion about a natural event, like drought, or about a market phenomenon, like inflation, once published online starts acquiring a socio-cultural texture and physiognomy, by occupying and diffusing in a virtual space and by progressively acquiring a set of roles in relation to the online ecosystem in which it is uploaded. These online ecosystems, despite being virtual, are culturalized, socialized, politicized and commoditized, similarly to other environments in which humans operate. That is, they are endowed with morphologies characterizing the aforementioned dimensions of human systems, in which contents and interactions are embedded. Therefore, online posts on social media go far beyond the bits used to store their digitized information and multimedia contents, and are embedded into complex social, cultural, political and economic contexts, to which they are linked and with which they are entangled.

Opinions, preferences, and expectations, as well as expectations of others’ opinions, preferences and expectations, can heavily impact human systems and their institutions, such as markets or governance bodies, affecting their functioning, stability and outcomes. For example, in a few hours, a short video posted on snapchat capturing a clash between police and activists protesting against the inaction of their government in relation to rising inequalities caused by extreme weather events, will be observed, probably in real time, by multiple journalists and bloggers around the globe, some of which will rapidly publish polarizing or clickbait online articles on the subject, which ‌could give rise to the outrage of a broader section of the population that will read them, which will probably express their reactions, feelings and expectations through another cascade of posts and comments on their preferred platforms, which, if not rapidly addressed by competent and reactive institutions, could produce broader social tensions, both in the digital and in the physical space, potentially creating a conflict spiral.

Private platforms gain, Twitter's X-changes reign, Blue checks cost, trust is lost, shifts in the digital domain.

 Most of the social media platforms, in which phenomena like the aforementioned spark and propagate, being private, aim to expand their profits and acquire new market quotas. Hence they operate for increasing actual and projected revenue streams. Recent changes of Twitter,  now also referred to as X, teach us that, in a few months, online platforms can drastically change in terms of business models, social interaction architectures, transparency, data access policies and search-engine algorithms. These changes may in turn affect the views, incentives to communicate and interaction patterns of their userbases. 

For example, among the many changes implemented by Twitter in recent months, one of the most significant is the decision to monetize the system of verified users, those with the blue checks in their profile, who now have also to pay a subscription for displaying this badge, which was previously a fair signal of the trustworthiness of Twitter sources, independently from users’ willingness and possibility to pay for it.

Among the many negative effects of this change, one, illustrated in this (https://www.washingtonpost.com/technology/2023/02/22/russian-propagandists-said-buy-twitter-blue-check-verifications) Washington Post’s article authored by Joseph Menn, is certainly a noteworthy example of the type of users that could be attracted by, and benefit the most from, these recent changes. The article’s title: “Russian propagandists are buying Twitter blue-check verifications”, speaks for itself.

Will this change, related to a new monetization strategy, destroy or bias the informative value of Twitter's blue check? If so, will the consumption of information generated by verified users change? and will Twitter users’ views be increasingly affected by “blue checked” partisan sources, which may want to afford the price of this subscription because of the benefits they derive from being labeled as verified sources? Is Twitter closing an eye, by allowing opaque profiles to display blue checks, only for a financial gain? Is this conflict of interest, due to Twitter’s incentive to maintain some opaque verified-user catchment areas, reshaping online debates in democratic countries?

Until mid-June 2023, with Twitter data freely accessible for academic research, a team of social scientists could have easily investigated these questions, to understand what is going on and to hence explain their findings to citizens, public institutions and civil society.  

(End of Chapter 1 – What effects had theTwitter/X changes? Discover them in Chapter 2!)