Thie deliverables D4.1 Communicating human-centric AI: overview of MUHAI communication activities, and D4.2 Disseminaitng human-centric AI: overview of MUHAI dissemination activities, offer extensive overviews of the activities that related to communication and dissemination of the Muhai project.

Discover tools and practices put in place by VIU with the support of all partners.

Narrative-based Understanding of Society, the MUHAI third and final volume is available in open access and can be downloaded at: https://zenodo.org/records/15093258

This book is the third open access volume in a series reporting results of the project. The first volume  surveyed how the notion of understanding is being discussed and treated in other human-centred research fields,  more specifically in social brain science, social psychology, linguistics, semiotics, economics, social history and medicine. The second volume explored these ideas with a number of concrete case studies that are all about understanding the world.

The present volume contains further case studies that now focus on understanding society. It introduces more tools for narrative understanding, and then reports on experiments to understand society through demography, through social media analysis, and through art. It also reports examples of generating social narratives from knowledge graphs. 

Full reference:

Steels, L., & van Harmelen, F. (2025). Narrative-based Understanding of Society. Zenodo. https://doi.org/10.5281/zenodo.15093258

As part of this milestone, two key knowledge graphs (KGs) were developed: the OKG includes social memories from contemporary data from twitter, and the MIRA-KG includes historical scientific data described in research articles. The OKG integrates tweet metadata and frames from social media discussions on inequality, helping to understand the discourse on this topic. Meanwhile, the MIRA-KG focuses on research related to socioeconomic history and health inequality, organizing research questions and findings from the scientific literature. Both resources are shared as open data on Zenodo, promoting accessibility and reuse by the research community.

The OKG 

The Observatory KG (OKG) is a knowledge graph which populates the Observatory Integrated Ontology (OBIO) (https://w3id.org/okg/obio-ontology/). The ontology integrates tweet metadata from tweets on inequality, and frames (meaning) extracted by parsing tweet texts.

Related publication: Blin, I., Stork, L., Spillner, L., & Santagiustina, C. R. M. A. (2023). OKG: A Knowledge Graph for Fine-grained Understanding of Social Media Discourse on Inequality (1.0.0) [Data set]. 

Article doi: https://doi.org/10.1145/3587259.3627557

Zenodo link: https://doi.org/10.5281/zenodo.10034210

The MIRA-KG 

The MIRA-KG is a knowledge graph created to get a grasp of research done in understanding our socioeconomic history. The resource populates the MIRA ontology (https://w3id.org/mira/ontology/), which links observations from society to the various steps of the research question derivation process. Specifically, the MIRA-KG is populated with research questions extracted from the scientific literature on health inequality.

Related publication: Stork, L., & Zijdeman, R. (2024). MIRA-KG: A Knowledge Graph of Hypotheses and Findings for Social Demography Research (Version 1.0.0) [Data set]. The Extended Semantic Web Conference (ESWC), Hersonissos, Greece. 

Article doi: https://doi.org/10.1007/978-3-031-60635-9_12

Zenodo link: https://doi.org/10.5281/zenodo.11316550

Are you interested in understanding how the Muhai outputs are built? Dive into codes and steps thanks to this helpful Canvas library.

Fully accessible at this link.

Narrative-based Understanding of Everyday Activities: An AI Cookbook, the second MUHAI volume was printed and presented to the Muhai project meeting in Venice, on the 12th of September.

The volume is the second of a series of three (please access the first one at this link).

The book ia edited by Luc Steels and Robert Porzel and focuses on theoretical research and concrete case studies about understanding everyday activities in the real world. The different papers push the state of the art in technologies needed to operationalize understanding: computational linguistics, semantic web technologies, cognitive robotics, mental and physical simulation, and generative AI. There are also contributions introducing data structures and strategies for integrating these components in the construction of integrative narrative networks and for more amenable user interfaces that provide better explanations, a crucial feature of human-centered AI. This volume also presents concrete benchmarks that exercise semantic forms of intelligence and reports on two operational examples meeting.

Fully accessible at this link!

Schematic Overview of the SIO Ecosystem by Laura Spillner

This project's output is the demonstration of a proactive intelligent observatory and analysis tools for social inequality. The observatory is based on both big data from historical archives as well as big data extracted from social media.

This interactive support tools for capturing and understanding narratives in socioe-conomic history research may be of use for researchers but also for policy makers. It allows the analysis and mapping of the public perception of social phenomena, highlighting correlations of interest. Moreover, it is helpful in tracing the large datasets created by social media.

The Social Inequality Observatory (SIO) was developed into an ecosystem of interconnected parts:

• knowledge stores of social inequality narratives retrieved from various textual datasets, for instance paper abstracts and tweet texts;

• an hybrid AI components for language understanding, narrative construction, analysis and contextualization as well as visualization of the results;

• different interfaces that users can interact with to explore different parts of the social inequality discourse (mainly scholarly and societal debate) and investigate the inequality narratives within them.

What are the three main assets that can be explored:

♦ Knowledge Store: The knowledge contained in the textual datasets is processed using a number of symbolic and statistical AI components, both existing ones and ones developed in the course of the MUHAI project. 

♦ HERMIONE: an interactive dashboard, we aim to bring greater visibility to these issues by providing a platform for exploring and understanding, in a user-friendly and visually appealing way, the perceptions of inequality in our society.

♦ MIRA:  which allows users to (i) turn research questions or paper abstracts into structured and interlinked data, (ii) explore pathways between research question variables to better understand social phenomena, (iii) plot geographical or temporal trends on maps, (iv) or explore the MIRA knowledge graph using HERMIONE’s co-occurrence network or fine-grained view.

Users will be able to develop additional interfaces by querying our knowledge graphs, by developing their own methods or by integrating the AI components published by the MUHAI project. All three parts are based on the MUHAI AI components, which will be published in one library as part of the CANVAS deliverable in the final project stage.

The MUHAI consortium (in particular the Sony Computer Science Laboratories-Paris, the VUA, the UHB and the VIU) are involved in this research process. 

Are you ready to visit the observatory?

Access it

The first Muhai volume, Foundations for Meaning and Understanding in Human-centric AI is the first of three, composing the Muhai book. It retraces the existing narrative-centric studies in order to identify the most promising research streams for tomorrow’s AI.  From conceptual foundations of human-centric AI to pragmatics of language, touching narratives in economic and social affairs, historical sciences, clinical trials, social neurosciences and art interpretation the human narratives are explored in their relationship with AI.

The volume is available at: https://zenodo.org/records/6666820.

Full Citation: Steels, L. (ed . ) . (2022). Foundations for Meaning and Understanding in Human-centric AI. In Foundations for Meaning and Understanding in Human-centric AI (1-6-2022, pag. 152) [Computer software]. Venice International University. https://doi.org/10.5281/zenodo.6666820

This benchmark for recipe understanding in autonomous agents aims to support progressing the domain of natural language understanding by providing a setting in which performance can be measured on the everyday human activity of cooking. Showing deep understanding of such an activity requires both linguistic and extralinguistic skills, including reasoning with domain knowledge. For this goal, the benchmark provides a number of recipes written in natural (human) English that should be converted to a procedural semantic network of cooking operations that can be interpreted and executed by autonomous agents. A system, which supports one-click installation and execution, is also included that can perform recipe execution tasks in simulation allowing both analysis and evaluation of predicted networks. The provided evaluation metrics are mostly simulation-based, because demonstrating deep understanding of recipes can be done by effectively taking all the appropriate actions required for cooking the intended dish.

The full benchmark has been made available standalone and as part of the Babel toolkit. Both options provide the same benchmark functionalities, but the Babel toolkit also provides the option of extending the system.

Download the benchmark: https://ehai.ai.vub.ac.be/recipe-execution-benchmark/

Discover  the benchmark in this video pill!

“The ultimate purpose of the social sciences is to furnish causal explanations of classes of observable events, which are, at least in part, generated by individual and collective agency/action.” [1] The aim of a digital assistant for social history research is therefore to support social scientists with the construction of such causal explanations for observable events, also theories or hypotheses. Social history researchers can then test these on various aspects of society, to see whether a newly found hypothesis holds. Since not many structured, easily accessible hypotheses exist in the social domain to learn from, we have focused on causal narratives in the medical domain first, since we have access to a dataset of ~4000 of these. The aim is to analyse this dataset, and transfer insights we gain to the social domain (given that they are transferable between domains). In this specification, we first briefly describe a few ways in which social history researchers discover and answer hypotheses, and potential things to look out for when developing a digital research assistant. Following, we briefly discuss the medical domain and hypothesis generation technologies we are developing in that area of research. 

Biomedical research: 
In the field of biomedicine, the common process to generate a new hypothesis that can be tested within a clinical trial, is to intervene in a given biochemical process with a specific treatment. This potential outcome framework has been used for a longtime fueled by new discoveries in the lab, e.g. the discovery of a new protein. As with social science research, the generation of a new hypothesis can includes the following steps:

Step 1. Protein-pathway discovery.
Example: ‘The discovery of a new oncogene participating in a cellular pathway’

Step 2. Drug discovery.
Example: ‘The development of a chemical molecule to target the new onco-gene.’

Step 3. Clinical trial.
Example: ‘A significant effect on tumor growth was found administering the chemical molecule to patients with liver cancer.’

Step 4. Drug repurposing by analogy.
Example: ‘The chemical molecule treats liver cancer, which resembles kidney cancer. Can the molecule treat kidney cancer too?

Hence, hypothesis generation can be fueled by a new scientific discovery such as the discovery of a new gene participating in a pathway, as well as by analogy, through already performed trials and their results. 

A digital assistant for biomedical research: 
Scientific discovery in the biomedical domain can greatly benefit from automated hypothesis generation, as finding new and interesting research questions is challenging and requires considerable background knowledge about trials, drugs, conditions and their various causal mechanisms. 

Two main requirements for automated scientific discovery: 

1. human should be able to follow the reasoning
2. human should be made aware of the potential biases in the data

The task is often formulated as a link prediction task, in which a new link is predicted between a disease and an existing treatment, such as insulin treats→ diabetes. Several  studies  argue  for  the  integration  of  a  model  with  structured  background knowledge about known cause and effect relationships within the problem domain, to support both the generation of hypotheses as well as their explanation. 

Explainable  link  prediction  methods  have  proved  very  successful  in  pointing out  new,  interesting  drug-treatment  pairs,  specifically  in  being  able  to  focus the attention of medical practitioners to those hypotheses that are explainable with current knowledge on biochemical processes. While these developments are paramount  in  producing  explainable  medical  AI,  such  hypotheses  are  subject to simplification. Bodily processes are complex in nature, and by reducing hypothesis generation to a single link prediction task, a system risks missing out on interesting hypotheses. For example: adults with diabetes mellitus as well asdiabetic ketoacidosis might require a completely different treatment than kids without diabetic ketoacidosis. Such a task can be formulated as a graph generation task, where one predicts not only a link between a drug and a disease, but the entirety of the hypothesis: age groups, symptoms, modes for drug delivery, and other. Even though explainable link prediction is a much researched topic, research into subgraph generation is  scarce,  and  the  research  that  exists  focuses  on  machine-learned methods that are often nontransparent in their reasoning. 

Social science/social history research: 

In the field of social history, the discovery of a causal narrative often arises first and foremost by the generation or discovery of a grand theory. Such grand theories can arise in a multitude of ways: from `rocking chair sociology’, to the discovery of certain patterns when zooming in on certain groups in society, be it inequality amongst people in a small town, or social cohesion amongst followers of a certain religion. Theories related to the latter therefore come about in a more fortuitous way. Here it is interesting to note that most finer-grained questions can be divided into three big questions or themes: those related to cohesion, inequality or rationalisation (the effect of technological developments on a society). 

When an interesting theory that is devised should be tested, or an interesting use case has come to light, the process of constructing causal narratives can be roughly subdivided into three sub-questions and their output.  Each following step ingests the output of the previous step: 

Start: either a theory, or a use case, e.g., the role of social cohesion can explain certain outcomes among social groups, or the suicide rates and religious beliefs of those living in town X have been recorded, respectively.

1. a descriptive question, e.g.,: are there more suicides among protestants than among catholics? Intended output: a temporally organised description of related events. 
2. an explanatory question, e.g.,: Can social cohesion explain the different statistics related to suicide among these different religious groups? Intended output: a causal narrative. 

Even though overarching ‘grand’ questions should remain the same, branching questions however are prone to grow into a certain direction. Knowledge on social history can therefore only lift one side of the curtain. 

A digital assistant for social science research: 
We argue that a digital assistant for scientific discovery in the social sciences or social history domain can aid in the data-driven generation of point 1. and 2. described in the section above. By ingesting structured data, such as is available at the international institute of social history (IISH), a digital assistant can, first and foremost, discover trends over time (longitudinal)  or among groups, to present to the researcher in question. An example of such a trend is described in point 1 above. Illuminating bias in datasets is an important component here, as bias limits the range of a certain hypothesis, for instance the hypothesis mentioned in the previous section could apply only to people that earn more than the marginal income. 

A digital assistant for hypothesis generation in the social sciences, be it social history or social science research in general, should take note of the following: 

Explainable. Humanities researchers increasingly turn their data into Linked Data[3,4],  interlinking their own data, but also to link social science data to knowledge from other domains available in the LOD cloud. 

References:
[1] Abell, P. (2009). History, case studies, statistics, and causal inference. European Sociological Review. https://doi.org/10.1093/esr/jcn072

Example literature related to a comparative question, as well as a data ecosystem supporting the search for causal narratives:

[2] van den Berg, N., van Dijk, I. K., Mourits, R. J., Slagboom, P. E., Janssens, A. A. P. O., & Mandemakers, K. (2021). Families in comparison: An individual-level comparison of life-course and family reconstructions between population and vital event registers. Population Studies, 75(1), 91–110. https://doi.org/10.1080/00324728.2020.1718186

[3] Hoekstra, R., Meroño-Peñuela, A., Rijpma, A., Zijdeman, R., Ashkpour, A., Dentler, K., Zandhuis, I., & Rietveld, L. (2018). The dataLegend ecosystem for historical statistics. Journal of Web Semantics, 50, 49–61. https://doi.org/10.1016/j.websem.2018.03.001

[4] ​​Zapilko, Benjamin, et al. "Applying linked data technologies in the social sciences." KI-Künstliche Intelligenz 30.2 (2016): 159-162.