AAAI-24 Tutorial and Lab List - AAAI (2024)

Sponsored by the Association for the Advancement of Artificial Intelligence
February 20-21, 2024 | Vancouver Convention Centre – West Building | Vancouver, BC, Canada

Half Day Tutorials

Quarter Day Tutorials

Half Day Labs

Quarter Day Labs

This tutorial, tailored for machine learning and applied math researchers and practitioners, focuses on emerging computational imaging applications, particularly addressing lesser-known inverse problems. Our objective is to shed light on crucial yet lesser-studied areas like snapshot compressive imaging and single-photon imaging, offering insights into their mathematical modeling, advancements, and the limitations of current solutions. In doing so, we also emphasize how AI/ML approaches are instrumental in addressing these challenges, enhancing traditional optimization-based methods to develop state-of-the-art algorithms in computational imaging.

Designed to cater to a diverse audience, this tutorial is ideal for individuals with a foundational understanding of basic linear algebra and convex optimization. While prior knowledge in deep learning and underdetermined linear inverse problems is advantageous, it is not mandatory. Our structured approach ensures that all attendees, regardless of their background, can effectively engage with the material and gain a comprehensive understanding of the subject matter.

The tutorial begins with a review of classic underdetermined linear inverse problems, highlighting optimization-based solutions and learning-based methods such as unrolled networks, solutions based on generative models, and autoencoders. We discuss the pros and cons of these algorithms and the challenges that remain unaddressed.

We then transition to three specific classes of emerging inverse problems in computational imaging:

1. Snapshot Compressive Imaging (SCI): This section delves into SCI systems innovative approach to capturing a 3D data cube from a single 2D projection. We will discuss applications of such systems, SCI recovery algorithms, and how AI/ML contributes to enhancing their efficiency and accuracy.
2. Single-Photon Imaging: This segment covers ultrafast imaging and non-line-of-sight imaging, showcasing how these techniques are revolutionizing our approach to capturing images.
3. Compressive Coherence Imaging: We explore imaging in the presence of speckle noise, discussing the intricacies and approaches to overcome these challenges.

By presenting a fundamental overview of each problem, along with existing solutions, we aim to pique attendees’ interest in these areas and provide them with the knowledge to effectively employ these solutions in their work. Our approach fosters a deeper understanding of a wide spectrum of inverse problems, ultimately contributing to advancements in the field of computational imaging.

For more detailed information about the tutorial, please visit our supplemental website: https://sites.google.com/view/aiforinverseproblems

The field of AI-generated content has experienced notable advancements recently, thanks to large language models and diffusion models that are capable of generating text and images. These developments have broadened applications across various domains, including text, image, video, and 3D object generation. Considering the increasing attention garnered by powerful generative models like ChatGPT for text and diffusion models for image synthesis, it is necessary for the AAAI community to fully explore these developments. This tutorial seeks to foster a deeper understanding of the field among conference attendees. Our tutorial will provide a comprehensive overview of AI-generated content, covering its foundations, frontiers, applications, and societal implications. It will cover the basics of large language models and diffusion models, as well as recent research and applications in this area. We will also discuss the societal concerns surrounding AI-generated content, including AI ethics and safety. By the end of the tutorial, attendees will have a better understanding of the current state of the field and the opportunities and challenges it presents. Our tutorial will be useful for researchers and practitioners interested in the application of AI-generated content to various domains. Attendees will gain insights into the latest techniques and tools for generating high-quality content and learn about the potential benefits and risks associated with this technology.

Keywords: AI Generated Content, Large Language Models, Diffusion Models, ChatGPT, Generative Models.

Website: https://sites.google.com/view/aigc-tutorial

Large language models (LMs) have achieved remarkable success in many language tasks. Recent works have also shown that knowledge of the world can emerge from large LMs, enabling them to assist decision-making for embodied tasks. However, the world knowledge exhibited by current large LMs is often not robust and cannot be grounded in physical environments without additional models. This hinders their ability to perform complex reasoning and planning tasks reliably. For example, in creating action plans to move blocks to a target state, GPT-4 achieves a significantly lower success rate compared to humans.

On the other hand, humans perform deliberate reasoning and planning based on the mental model of the world, also known as a world model (WM), which enables us to simulate actions and their effects on the world’s state. WMs encoding knowledge of the physical world can drastically improve the data efficiency and robustness of intelligent agents.

However, WMs were typically studied in reinforcement learning and robotics, areas conceptually distinct from problems studied in language modeling. This gap indicates new opportunities for connecting WMs and LMs to enhance LM capabilities in reasoning and planning in both embodied and general settings, and address the aforementioned limitations. Emerging studies on the intersection of WMs and LMs have demonstrated promising results. This tutorial aims to summarize and present a unified view of connecting WMs and LMs, highlighting the various opportunities for improved machine reasoning and planning based on large LMs through world modeling. We will review recent works on learning WMs and using them to further learn and perform embodied tasks. We will show how LMs can utilize external WMs to compensate for their lack of grounded world knowledge and how LMs themselves can learn world models from embodied experiences beyond text data, and use these internal WMs to guide complex reasoning.

This tutorial is geared towards researchers and practitioners interested in imposing requirements to machine learning (ML) systems, such as fairness, robustness, and safety. Typically, these statistical, data-driven constraints are induced by combining the learning objective and requirement violation metrics into a single training loss. To guarantee that the solution satisfies the requirements, however, this approach requires careful tuning of hyperparameters (penalty coefficients) using cross-validation, a computationally intensive and time consuming process. Constrained learning incorporates requirements as statistical constraints rather than by modifying the training objective. In this tutorial, we provide an overview of theoretical and algorithmic advances from the past 5 years that show when and how it is possible to learn under constraints and effectively impose constraints on ML systems, both during training and at test time. Specifically, we explore the role and impact of different types of requirements in supervised learning, robust learning, and reinforcement learning (RL). First, we introduce new non-convex duality results that yield generalization guarantees for constrained supervised learning. We also use these results to derive practical algorithms to tackle these problems, despite their non-convexity. We then leverage these advances to obtain algorithms for robust learning capable of achieving better trade-offs between nominal and adversarial accuracy. Finally, we develop a parallel theory for constrained RL, showing that it is strictly more expressive than its unconstrained counterpart. Throughout the tutorial, we illustrate the effectiveness and flexibility of constrained learning in a diverse set of applications from robust image classification, federated learning, learning under invariance, and safe RL. Ultimately, this tutorial provides a general tool that can be used to tackle a variety of problems in ML and sequential decision-making.

Organizers

As AI technologies are increasingly deployed in real world applications, notably in the form of search engines, recommender systems, and conversational assistants, how to evaluate such technologies in the context of interactive intelligent system applications has emerged as an urgent challenge for both practitioners who deploy AI products and researchers.

Research communities have so far mostly relied on test collections to perform reproducible experiments, but such an evaluation methodology cannot be used to evaluate interactive intelligent systems, whose utility must be assessed by users via interactions with the system. To tackle this challenge, researchers have proposed and developed an evaluation methodology based on user simulation, where the idea is to simulate a real user using an intelligent agent that can mimic a user’s decisions when interacting with an AI system and evaluate an AI system by having the system interact with such an artificial user and measuring the perceived utility and cost/effort from a user’s perspective for finishing a task. The work on user simulation for evaluating intelligent systems has so far been mostly done in applied AI communities, notably Information Retrieval, Recommender Systems, and World Wide Web. The goal of this tutorial is to provide a systematic review of this topic and discuss many interesting novel AI-related research challenges for AAAI attendants, allowing them to learn about the major ideas, frameworks, models, and algorithms for both building user simulators and using simulators to evaluate an interactive system, as well as important future research directions.

This introductory tutorial primarily targets graduate students, academic researchers, and industry practitioners who are interested in learning about how AI techniques can be applied to build user simulators and/or how user simulation can be used to evaluate interactive AI systems. Since the question of how to accurately evaluate interactive intelligent systems is important to both practitioners who would like to assess the utility of their product systems and researchers who would like to know whether their new algorithms are truly more effective than the existing ones, we expect our tutorial to be broadly appealing to many participants of AAAI.

For more details seehttps://usersim.ai/aaai2024-tutorial/

Combinatorial optimization problems are ubiquitous in our lives. They show up in the forms of matching problems (e.g., assigning junior doctors to hospitals), scheduling problems (e.g., radiation therapy), logistics problems (e.g., visiting nurses), and many more. In many cases, these problems are also notoriously hard (often NP-hard, or even worse). Still, thanks to tremendous progress over the last decades, we now have access to sophisticated algorithms that can solve these problems in practice.

Unfortunately, it turns out that, due to their immense complexity, these (solving) algorithms often contain subtle bugs.In this tutorial, we give an overview of the most successful approach to dealing with this issue, namely proof logging, meaning that solvers aren’t allowed to just claim an answer to a problem: they’re expected to also produce an independently verifiable proof that backs up this claim. In the field of Boolean Satisfiability, this has done wonders for solver reliability, and has also helped with social acceptability of computer-generated mathematical results. What if we could do the same for the more general field of constraint programming, producing an auditable solving process where people whose lives or livelihoods are affected by a computer’s decision would no longer have to resort to hoping that the system is free of bugs?

We will start this tutorial by explaining what a proof really is, and what it means for an algorithm to certify the correctness of its answers by using proof logging. We will give a brief overview of the (extended) resolution and DRAT proof systems used in SAT proof logging. Then we will look at what is needed to bring proof logging to a broader range of solving algorithms, starting with some subgraph-finding algorithms, and moving towards a full CP solver with multiple global constraints and even some reformulation. We will show, by example, what you need to do to use this technology in your own algorithms, and how this relates to the underlying proof methods.
We’ll finish by discussing how proof logging can also support advanced solving techniques such as symmetry breaking. Surprisingly, all of this can be done in a simple proof format known as “cutting planes with strengthening” that does not know anything about non-binary variables, global constraints, graphs, groups, or symmetries.

Large Language Models (LLMs) have showcased immense potential in generating human-like text as demonstrated by numerous studies. Despite their remarkable capabilities, LLMs such as ChatGPT can occasionally err in maintaining factual accuracy or logical consistency. They might inadvertently generate content that is harmful or offensive, and they are unaware of events that have occurred post their training phase. The challenge at hand, quite intuitively, remains how to effectively update these LLMs or rectify their errors without resorting to a complete retraining or continuous training process, both of which can be significantly resource-intensive and time-consuming. To this end, the notion of knowledge editing for LLMs has been proposed which provides an efficient way to modify the model’s behavior, particularly within a specified domain of interest, without compromising its performance on other inputs.

We will initiate this tutorial by defining the tasks involved in knowledge editing for LLMs, and introducing evaluation metrics and benchmark datasets. Subsequently, we will provide an overview of various knowledge editing methodologies. Initially, our focus will be on methods that preserve the parameters of LLMs. These techniques function by adjusting the model’s output for certain instances, achieved by integrating an auxiliary network alongside the original, untouched model. We will then transition to methods that modify the parameters of LLMs, which aim to alter the model parameters responsible for undesirable outputs. Throughout the tutorial, we aim to share insights gleaned from the diverse communities engaged in knowledge editing research and introduce open-sourced tools EasyEdit. Further, we will delve into potential issues as well as opportunities associated with knowledge editing for LLMs, with the goal of imparting valuable insights to the community. All tutorial slides will be available athttps://github.com/zjunlp/KnowledgeEditingPapers.

Learning with multiple objectives emerges frequently as a new unified learning paradigm from recent machine learning problems such as learning under fairness and safety constraints; learning across multiple tasks including multi-task learning and meta-learning; learning across multiple agents including federated and multi-agent learning; and, learning with hierarchical games including incentive designs, reward shaping, and Stackelberg game. This tutorial will introduce principled frameworks that cover bilevel optimization for learning under two objectives with preference, and multi-objective optimization for learning with multiple objectives without preference, and their combinations. Efficient algorithms will be presented and recent advances on optimization and generalization theory will be introduced. We will highlight how we can apply those algorithms and theories to multi-task learning and learning with Markov games. Upon completion, the audience is expected to gain the necessary knowledge to effectively perform learning tasks with multiple objectives in and beyond the afore-mentioned applications.

External link to the website: https://sites.google.com/view/mol-tutorial/home

Machine learning techniques have made significant strides, leading to a wide array of general and specialized models that fulfill practical needs. As a result, model reuse has emerged as a crucial technique, empowering developers to harness the power of pre-trained models (PTMs), including small models and even large language models, to enhance the performance and efficiency of their target machine learning systems. These PTMs encapsulate valuable inductive biases that are helpful for target tasks, and well-designed reuse strategies are capable of extracting knowledge from a given model, pushing them beyond their original scope, and facilitating diverse machine learning tasks.

This tutorial establishes connections between the widely applied notion of model reuse across various domains and presents a modern taxonomy that covers the full stages of leveraging PTMs, namely in data preparation, architecture design, model training, and model inference. Besides providing a comprehensive overview of typical model reuse methods, the tutorial also explores extensions of basic model reuse notions, such as the active selection of PTMs from a model zoo, promising applications of model reuse, and the challenges, limitations, and trends in model reuse.

Machine learning (ML) is spurring a transformation in the computational sciences by providing a new way to build flexible, universal, and efficient approximations for complex high-dimensional functions and functionals. One area in which the impact of these new tools is beginning to be understood is the physical sciences, where traditionally intractable high-dimensional partial differential equations are now within reach. This tutorial will explore how developments in ML complement computational problems in the physical sciences, with a particular focus on solving partial differential equations, where the challenges of high-dimensionality and data acquisition also arise.

The first important example this tutorial will cover is using Deep Learning Methods for solving high-dimensional PDEs, which have wide application in variational rare events calculations, many-body quantum systems, and stochastic control. Another challenge covered by this tutorial that researchers often face is the complexity or lack of specification of the models they are using when performing uncertainty quantification. Thus another line of research aims to recover the underlying dynamic using observational data.

This tutorial will introduce the well-developed methods and theories for using machine learning in scientific computing. We will first discuss how to incorporate physical priors into machine learning models. Next, we will discuss how these methods can help to solve challenging physical and chemical problems. Finally, we will discuss the statistical and computational theory for scientific machine learning. In this tutorial, we will not focus on the technical details behind these theories, but on how they can help the audience to understand the challenges of using machine learning in differential equation applications and to develop new methods for addressing these challenges.

More Information to Come

The overarching goal of this tutorial is twofold: The first aim is to conduct a comprehensive assessment of the latest advancements in the gradient-free learning paradigm, also referred to as zeroth-order machine learning (ZO-ML). This involves an exploration of the theoretical and methodological foundations that support ZO-ML. The second goal is to illustrate the effective integration of ZO-ML techniques with emerging ML/AI applications. This step aims to bridge the theoretical and practical aspects of ZO-ML, demonstrating its potential to overcome design limitations in current foundation model (FM)-oriented applications.

Format

The tutorial comprises five parts: an introduction to ZO-ML, including its mathematical foundations and a comparison with first-order methods; an exploration of the foundational algorithms, properties, and scaling techniques of ZO-ML; a focus on ZO-ML applications in various AI fields; a practical demonstration session showcasing ZO-ML tools, benchmarking, and live demos in large language models and adversarial defense; and a concluding segment with key takeaways, future perspectives, and additional resources for deepening knowledge in ZO-ML.

Graph learning has emerged as a prominent area of interest in both academic and industrial circles, aiming to model complex graph-structured data. While existing methods primarily concentrate on leveraging node features and topological structures, there is untapped potential in enhancing graph learning through auxiliary knowledge. This auxiliary knowledge refers to valuable information that can be obtained, extracted, or learned from resources beyond the provided node features and structures. Exploring the realm of knowledge-enhanced graph learning (KEGL) is crucial for improving learning capabilities. In this tutorial, our focus is on showcasing state-of-the-art techniques in KEGL. We delve into the diverse sources of knowledge that can benefit graph learning, including internal perception, external sources, human expertise, and knowledgeable models. Additionally, we provide insights into real-world applications and engage in discussions about future research directions.

Graph Neural Networks (GNNs) have gained significant attention in recent years due to their ability to model complex relationships between entities in graph-structured data such as social networks, protein structures, and knowledge graphs. However, due to the size of real-world industrial graphs and the special architecture of GNNs, it is a long-lasting challenge for engineers and researchers to deploy GNNs on large-scale graphs, which significantly limits their applications in real-world applications. In this tutorial, we will cover the fundamental scalability challenges of GNNs, frontiers of large-scale GNNs including classic approaches and some newly emerging techniques, the evaluation and comparison of scalable GNNs, and their large-scale real-world applications. Overall, this tutorial aims to provide a systematic and comprehensive understanding of the challenges and state-of-the-art techniques for scaling GNNs. The summary and discussion on future directions will inspire engineers and researchers to explore new ideas and developments in this rapidly evolving field. The website of this tutorial is available at https://sites.google.com/ncsu.edu/largescalegnn/home/aaai_2024.

Keywords

Graph Neural Networks; Large-scale Graphs; Scalability

Descriptions

This tutorial represents a notable achievement as it offers a comprehensive overview of techniques designed for largescale machine learning on graphs, encompassing both theoretical foundations and practical applications. It delves into past and recent research endeavors aimed at enhancing the scalability of Graph Neural Networks (GNNs) and explores their diverse potential use cases. Specifically, this tutorial will provide a thorough examination of the theory and algorithms for a wide array of methods, including both existing and emerging ones, ensuring that attendees leave with a strong theoretical foundation and a profound grasp of these methods. Furthermore, this tutorial will also highlight their applications in industry, bridging the gap between theory and real-world practice. This comprehensive coverage ensures that participants are not only well-versed in specific. All rights reserved. methods but also informed about the latest developments in the field and how to apply them in real-world scenarios. The prerequisite knowledge for our proposed tutorial expects that attendees have a foundational understanding of machine learning and deep learning concepts. Familiarity with linear algebra, calculus, and probability theory, as these mathematical principles are fundamental to GNNs algorithms. Moreover, a basic understanding of graph theory and graph data structures is also preferred. Although we will offer an introductory overview of Graph Neural Networks (GNNs) during the tutorial, participants with prior knowledge in these areas will be better prepared to engage with advanced GNN concepts. Furthermore, possessing expertise in specific application domains where GNNs are frequently utilized, such as social networks, recommendation systems, or natural language processing, will prove beneficial for attendees interested in practical applications.

Machine learning has become a powerful tool for discrete optimization. Discrete optimization algorithms have an incredible array of applications, including routing, manufacturing, planning, finance, and many others. The problems that, for example, a shipping company must solve to route its trucks will change daily, but not drastically: although demand and traffic will vary, the underlying road network will remain the same. This means that there is likely underlying structure that can be uncovered with the help of machine learning to optimize algorithm runtime on future problems. Since discrete optimization algorithms are used so widely, and since discrete optimization problems are so difficult to solve, this use of machine learning has the potential to make a significant impact in industry and scientific research.

This tutorial will cover how machine learning can be used within the discrete optimization pipeline from many perspectives, including how to design novel combinatorial algorithms with machine-learned modules and configure existing algorithms’ parameters to optimize performance. Topics will include both applied machinery (such as graph neural networks and reinforcement learning) as well as theoretical tools for providing provable guarantees.

I will only assume a background in machine learning. The tutorial will cover all other relevant background, such as graph neural networks and statistical learning theory tools.

This tutorial bridges two dynamic and evolving fields in AI: large language models (LLMs) and privacy-preserving AI. As we witness the proliferation of AI models, including LLMs, across various domains, the imperative to ensure privacy and security has never been more critical. We seek to unite these disciplines by introducing researchers and practitioners in the LLM space to cutting-edge privacy-preserving techniques that have the potential to revolutionize high-stake AI systems.

As the deployment of AI models in sensitive areas like healthcare, finance, and communication continues to grow, safeguarding privacy becomes paramount. Privacy-preserving techniques, particularly when applied to large language models, represent a relatively new and rapidly evolving field. This course introduces the LLM community to practical security approaches and novel privacy-preserving methodologies (such as federated learning, differential privacy, and secure multi-party computation) that are not yet widely explored within the LLM frontier research with its unique pre-training, fine-tuning and deployment requirements, but have immense potential for high-stake applications, such as healthcare, forensics, and justice.

With the increasing scrutiny on data privacy regulations and ethical AI practices, the tutorial addresses a topic that is at the forefront of discussions in both the LLM and AI security communities. Understanding and implementing privacy-preserving methods is essential for compliance with evolving regulations and maintaining public trust. Following the upcoming book in early 2024 by the author on the same topic, the course provides actionable insights and tools that A( professionals can immediately apply to their work. Attendees will gain both the methodological understanding and hands-on coding experience with techniques that enable the responsible deployment of large language models in real-world high-stake applications.

Humans possess unique capabilities for forming concepts from experience, such as acquiring and revising them in an incremental, unsupervised, and cumulative way. Creating machine learning approaches that can realize these, and other, human-like capabilities remains a tantalizing goal. This tutorial delves into this intriguing area, showcasing the Cobweb family of approaches, which provide a foundation for human-like machine learning. These approaches adopt a probabilistic formalism to support robust incremental and unsupervised concept learning. Moreover, Cobweb models support prediction along multiple dimensions simultaneously, making them more flexible than traditional supervised learning models.

This session will present Cobweb’s core methodology for learning and performance, discuss several variants and extensions, and demonstrate its unique capabilities across several real-world applications, such as tabular data clustering and prediction and language and vision modeling. Our interactive tutorial format will walk participants through several examples, offering hands-on guidance to obtain and run code for each. We aim to bridge the gap between theory and practice, offering insights into how Cobweb’s human-like learning methodology can lead to new machine learning advancements.

Our tutorial invites participants from diverse backgrounds and application areas that are keen to explore an innovative new concept learning paradigm. It is designed for participants with a basic background in computer science and AI/ML; although, participants with deeper knowledge in these areas will find the tutorial beneficial too.

For more details see https://humanlikelearning.com/aaai24-tutorial/.

In this tutorial we will present the “map of elections” framework for conducting experiments and analyzing election data in computational social choice. The idea of the framework is to take a set of elections, compute distances between them, and present them as points on a plane, whose distances resemble the distances between the elections. First, we will present the main components of the framework, including different ways of computing distances between elections, sources of election data, and algorithms for embedding elections in 2D space. Second, we will show a number of use cases of the framework, including planning and analyzing experiments, as well as analyzing synthetic and real-life data. Finally, we will show various extensions of the framework.

The target audience of this tutorial are researchers working on all aspects of computational social choice, with a particular focus on researchers studying elections and on early-stage researchers. In principle, the attendees can come without prior knowledge (of computational social choice), as all relevant notions will be explained. However, to make the most out of the tutorial, it will be helpful for the attendees to have a basic understanding of ordinal elections and voting rules used to select winners (such as the Plurality rule, the Borda rule, the notion of a Condorcet winner etc.). In some parts of the tutorial, it will be helpful to have a minimal background regarding multiwinner and approval elections (including the notion of justified representation and cohesive groups).

https://home.agh.edu.pl/~pragma/tutorials/aaai24/

Neural network driven applications like ChatGPT suffer from hallucinations where they confidently provide inaccurate information. A fundamental reason for this inaccuracy is the lack of robust measures that are applied on the underlying neural network predictions. In this tutorial, we identify and expound on three human-centric robustness measures, namely explainability, uncertainty, and intervenability, that every decision made by a neural network must be equipped and evaluated with. Explainability and uncertainty research fields are accompanied by a large body of literature that analyze decisions. Intervenability, on the other hand, has gained recent prominence due its inclusion in the GDPR regulations and a surge in prompting-based neural network architectures. In this tutorial, we connect all three fields using gradient-based techniques to create robust machine learning models.

The goal of the tutorial is threefold: 1) Decompose modern large-scale neural network robustness into three manageable and human-centric measures, 2) Probabilistically define post-hoc explainability, uncertainty, and intervenability measures, and 3) Compute the three measures as a function of the input, network and output with a focus on real life applications in medical, and seismic domains. The tutorial is composed of four major parts. Part 1 discusses some recent surprising results regarding training neural networks with out-of-distribution (OOD) data, the conclusions of which are that it is not always clear when and how to use OOD data during training. This motivates the need for formal and human-centric measures of robustness at Inference. Part 2 introduces the basic mathematical framework for each one of Explainability, Uncertainty, and Intervenability. We consider the relationships between them and show that interventions on variance decomposition of predictive uncertainty is an evaluation measure used for explainability. Part 3 illustrates the relationship between Explainability and Uncertainty in real world applications of biomedical and geophysics image analyses. Part 4 discusses intervenability as a function of uncertainty for the case of prompting and non-causal interventions. The detailed subtopics, tutorial outline, and suggested reading are present at https://alregib.ece.gatech.edu/aaai-2024-tutorial/.

This tutorial is intended for graduate students, researchers, engineers, and data scientists working in different topics related to visual information processing, machine learning, robust machine learning, and explainable AI. The audience are expected to have a basic understanding of neural networks and robustness applications including image recognition and detection.

To make effective decisions, it’s important to have a thorough understanding of the causal connections among actions, environments, and outcomes. This tutorial aims to surface three crucial aspects of decision making through a causal lens: 1) the discovery of causal relationships through causal structure learning, 2) Understanding the impacts of these relationships through causal effect learning, and 3) applying the knowledge gained from the first two aspects to support decision-making via causal policy learning. This tutorial aims to offer a comprehensive methodology and practical implementation framework by consolidating various methods in this area into a Python-based collection. This tutorial will provide a unified framework for you to understand areas including causal inference, causal discovery, randomized experiments, dynamic treatment regimen, bandits, reinforcement learning, and so on. This tutorial is based on an online book with an accompanying Python package (in progress; collaboration is welcomed).

Large Language Models (LLMs, or n-gram models on steroids) that have been trained originally to generate text by repeatedly predicting the next word in the context of a window of previous words, have captured the attention of the AI (and the world) community. Part of the reason for this is their ability to produce meaningful completions for prompts relating to almost any area of human intellectual endeavors. This sheer versatility has also led to claims that these predictive text completion systems may be capable of abstract reasoning and planning. In this tutorial we take a critical look at the ability of LLMs to help in planning tasks–either in autonomous modes, or in assistive modes. We are particularly interested in characterizing these abilities–if any–in the context of problems and frameworks widely studied in the AI planning community. The tutorial will both point out the fundamental limitations of LLMs in generating plans that will normally require resolving subgoal interactions with combinatorial search, and also show constructive uses of LLMs as complementary technologies to the sound planners that are developed in the AI Planning community. In addition to presenting our own work in this area, we provide a critical survey of many related efforts, including by researchers outside of the planning community.

Tutorial Website: http://rakaposhi.eas.asu.edu/llm-planning-tutorial.html

In today’s interconnected world, systems with large populations of strategic agents are ubiquitous, from distributed robotics, advertisem*nt and marketing, network routing, self-driving cars, to macroeconomics and finance. In the past decade, significant progress has been made, on both the conceptual and the computational aspects, unlocking the potential for more efficient and more robust techniques in analyzing and learning large stochastic games. In this tutorial, after introducing a number of motivating examples and related concepts, we will survey several approximation approaches, including mean field techniques and optimization algorithms, to study extremely large populations of rational agents, along with several applications and examples. We will then review recent advances on the computational side with a focus on state-of-the-art learning and optimization algorithms. Last, we will conclude with some challenges and future directions. Throughout the tutorial, we will present online open software packages and codes to illustrate relevant techniques.

The main modeling framework will be large games with stochastic dynamics, in which many strategic agents optimize a reward while interacting with each other. The core concepts are the notion of Nash equilibrium and Pareto optimum. In terms of methods, we will focus on learning and optimization-based algorithms: starting with simple methods, we will then show the state-of-the-art algorithms based on optimization, deep learning, and reinforcement learning.

The target audience is anyone interested in learning about optimization and games with large populations, their applications, and related solution approaches and learning algorithms. We expect some familiarity with Markov Decision Processes and basic optimization concepts, but we do not expect any specific expertise beyond the familiarity that would be expected of the usual AAAI attendee. The participants will learn how to phrase various problems in the framework of large stochastic games, and will gain knowledge in solution methods based on machine learning. They will also be able to apply library codes to solve their practical problems. The topic of this tutorial has gained exponential momentum in the machine learning community over the past few years, and it will also be of interest for the general audience of AAAI. Our aim is to make the topic more accessible to attendees so they will be able to contribute to this field and utilize these tools and algorithms to solve problems relevant to their fields.

Knowledge should not be accessible only to those who can pay” said Robert May, chair of UC’s faculty Academic Senate. Similarly, machine learning should not be accessible only to those who can pay. Thus, machine learning should benefit to the whole world, especially for developing countries in Africa and Asia. When dataset sizes grow bigger, it is laborious and expensive to obtain perfect data (e.g., clean, safe, and balanced data), especially for developing countries. As a result, the volume of imperfect data becomes enormous, e.g., web-scale image and speech data with noisy labels, images with specific noise, and long-tail-distributed data. However, standard machine learning assumes that the supervised information is fully correct and intact. Therefore, imperfect data harms the performance of most of the standard learning algorithms, and sometimes even makes existing algorithms break down. In this tutorial, we focus on trustworthy learning when facing three types of imperfect data: noisy data, adversarial data, and long-tailed data.

Prerequisites for Audiences:

Although we will introduce the basics of knowledge required, it is better to have some knowledge in linear algebra, probability, machine learning, and artificial intelligence. The emphasis will be on the intuition behind all the formal concepts, theories, and methodologies. The tutorial will be self-contained in a high level. Besides, audiences who are familiar with the basic robustness under noise, adversary and imbalance will find the comprehensive advances of areas.

Website:

https://tmlr-group.github.io/tutorials/aaai2024.html

Quarter Day Tutorials

Molecular representation pretraining is critical in various applications for drug and material discovery. Along this research line, most existing work focuses on pretraining on 2D molecular graphs. Meanwhile, the power of pretraining on 3D geometric structures has been recently explored. In this tutorial, I would like to start the introduction to molecule geometric representation methods (group invariant and equivariant representation) and self-supervised learning for pretraining. After this, I will combine these two topics and comprehensively introduce geometric pretraining for molecule representation, discussing the most recent works in detail (GraphMVP, GeoSSL, and MoleculeSDE).

Graph machine learning has been extensively studied in both academia and industry. Although booming with a vast number of emerging methods and techniques, most of the literature is built on the in-distribution (I.D.) hypothesis, i.e., testing and training graph data are sampled from the identical distribution. However, this I.D. hypothesis can hardly be satisfied in many real-world graph scenarios where the model performance substantially degrades when there exist distribution shifts between testing and training graph data. To solve this critical problem, out-of-distribution (OOD) generalization on graphs, which goes beyond the I.D. hypothesis, has made great progress and attracted ever-increasing attention from the research community. This tutorial is to disseminate and promote the recent research achievement on out-of-distribution generalization on graphs, which is an exciting and fast-growing research direction in the general field of machine learning and data mining. We will advocate novel, high-quality research findings, as well as innovative solutions to the challenging problems in out-of-distribution generalization and its applications on graphs.

This tutorial will be highly accessible to the whole machine learning and data mining community, including researchers, students and practitioners who are interested in this topic. The tutorial will be self-contained and designed for introductory and intermediate audiences. No special prerequisite knowledge is required to attend this tutorial.

For more details, please refer tohttps://ood-generalization.com/aaai2024Tutorial.htm.

Disentangled Representation Learning (DRL) aims to learn a model capable of identifying and disentangling the underlying factors hidden in the observable data in representation form.

The process of separating underlying factors of variation into variables with semantic meaning benefits in learning explainable representations of data, which imitates the meaningful understanding process of humans when observing an object or relation. As a general learning strategy, DRL has demonstrated its power in improving the model explainability, controllability, robustness, as well as generalization capacity in a wide range of scenarios such as computer vision, natural language processing, data mining, etc. In this tutorial, we will disseminate and promote the recent research achievements on disentangled representation learning as well as its applications, which is an exciting and fast-growing research direction in the general field of machine learning.

This tutorial consists of four parts. We will first give a brief introduction on DRL, followed by discussions on learning strategies and applications of disentangled representation learning, covering disentangled graph representation learning, disentangled representation learning for multi-modalities, and disentangled representation for recommendation, etc. We finally share some of our insights on the trending and future directions for disentangled representation learning, such as disentangled representation learning and large pretrained models. This tutorial will be highly accessible to both the general AI community, including researchers, students and practitioners who are interested in representation learning for AI, especially for explainable and controllable AI. The tutorial will be self-contained and designed for introductory and intermediate audiences.

In recent years, some new learning paradigms have been proposed to address various challenges in practical applications.
Of particular interest in this tutorial is the learning paradigm that can be formulated as a stochastic nested optimization (SNO) problem
because it covers a wide range of emerging machine learning models, such as model-agnostic meta-learning, imbalanced data classification models, contrastive self-supervised learning models, neural architecture search, etc. More specifically, this tutorial will focus on two instances of SNO: stochastic compositional optimization (SCO) and stochastic bilevel optimization (SBO).
The nested structure in these two classes of optimization problems brings unique challenges for their distributed optimization.
A series of new techniques have been developed to address them in the past few years. However, these recent advances have not been disseminated to broad audiences.
Thus, this tutorial aims to introduce the unique challenges, recent advances, and practical applications of distributed SCO and distributed SBO, with a special focus on federated learning and decentralized learning settings. The audience will benefit from this tutorial by mastering the understanding of distributed SCO/SBO algorithms and applying them to real-world applications.

Website: https://www.cst.temple.edu/~tuo14379/tutorial_aaai2024.html

This tutorial aims to bring together AI researchers, data scientists, and industry practitioners to explore out-of-distribution generalization challenges in time series. Time series data are prevalent in many domains, including retail, finance, healthcare, and environmental monitoring, serving as a basis for numerous applications. A major challenge in time series studies is the presence of dataset shifts, where the statistical properties of data may vary due to data collection from various sources, different locations, or conditions. Out-of-distribution generalization is the task where models should generalize to new, unseen scenarios/domains by learning from observed, seen scenarios. In this tutorial, we embark on a journey to explore the confluence of generalization and time series analysis, revealing insights into how out-of-distribution generalization techniques can be harnessed to tackle challenges in modeling time series data. We begin by laying a strong foundation of time series problems and out-of-distribution generalization. Then, we delve into the challenges, problems, methods, and evaluation in improving out-of-distribution generalization in the time series domain. Recent research findings in this emerging field and their implications will be unveiled, providing participants with the tools to advance their work. Finally, we turn toward the future, discussing emerging trends, open research questions, and the rich set of possibilities in this field.

We are seeing a significant growth in the Internet of Things (IoT) and mobile applications which are based on predictive analytics over time-series data collected from various types of sensors and wearable devices. Some important applications include smart home automation, mobile health, smart grid management, and finance. Traditional machine learning and deep learning has shown great success in learning accurate predictive models from time-series data. However, safe and reliable deployment of such machine learning (ML) systems require the ability to be robust to adversarial/natural perturbations to time-series, to detect time-series data which does not follow the distribution of training data, aka out-of-distribution (OOD) detection.

Most of the prior work on adversarial robustness for deep models is focused on the image domain and natural language domain to a lesser extent. On the other hand, business data such as stock prices and financial transactions, IoT data, and health data rely heavily on a time-series modality for AI-powered analysis and predictions. Transferring prior work on adversarial robustness to these domains has several caveats due to the different characteristics of modalities. Time-series domain poses unique challenges (e.g., sparse peaks, fast oscillations) that are not encountered in both image and natural language processing domains. Therefore, prior approaches are not applicable as they don’t capture the true similarity between time-series instances.

This tutorial will cover recent advances in adversarial robustness and certification for time-series domain using appropriate distance measures (e.g., dynamic time warping); min-max optimization algorithms to train robust ML models for time-series domain; OOD detection methods using deep generative models with application to generation of synthetic time-series data; and threats of adversarial attack on multivariate time-series forecasting models and viable defense mechanisms. This tutorial will also cover the real-world applications that require reliable and robust time-series analytics such as classification in human activity monitoring for smart health, and regression/forecasting in financial data.

The target audience of this tutorial includes (1) General AI researchers and graduate students who will learn about principles, algorithms, and outstanding challenges to explore the frontiers of robust time-series ML and its real-world applications; (2) Time-series ML researchers who will learn about the complete landscape of robustness and trustworthiness to gain breadth and learn about outstanding challenges in the frontiers; and (3) Industrial AI researchers and practitioners who will be apply the learned knowledge for solving applications including mobile health, smart homes, and smart grid.

Most real-world graphs constantly grow or evolve with potential distribution shifts. Classical graph learning models, however, typically assume graphs to be static and suffer from catastrophic forgetting when new types of nodes and edges (or graphs) continuously emerge. Therefore, investigating how to constantly adapt a graph learning model to new distributions/tasks in the growing graphs without forgetting the previously learned knowledge, i.e. Continual Graph Learning (CGL), is becoming increasingly important in various real-world applications, e.g., social science, biomedical research, etc. Due to the existence of complex topological structures, CGL is essentially different from traditional continual learning on independent data without topological connections (e.g., images). Challenges in CGL include the task configuration in different types of graphs, preservation of the previously learned topology, properly handling the concept drift caused by the topological connections, etc. In this tutorial, we will introduce this newly emerging area – Continual Graph Learning (CGL). Specifically, we will (1) introduce different continual graph learning settings based on various application scenarios, (2) present the key challenges in CGL, (3) highlight the existing CGL techniques and benchmarks, and (4) discuss potential future directions.

Prerequisite:

Our tutorial targets the audience with a basic knowledge of machine learning and deep learning, e.g., how to construct and optimize neural networks. The knowledge of continual learning or CGL is not required.

Website:

https://queuq.github.io/CGL_AAAI2024/

This tutorial focuses on curriculum learning (CL), an important topic in machine learning, which gains an increasing amount of attention in the research community. CL is a learning paradigm that enables machines to learn from easy data to hard data, imitating the meaningful procedure of human learning with curricula. As an easy-to-use plug-in, CL has demonstrated its power in improving the generalization capacity and convergence rate of various models in a wide range of scenarios such as computer vision, natural language processing, data mining, reinforcement learning, etc. Therefore, it is essential introducing CL to more scholars and researchers in the machine learning community. However, there have been no tutorials on CL so far, motivating the organization of our tutorial on CL at AAAI 2024.

To give a comprehensive tutorial on CL, we plan to organize it from the following aspects: (1) theories, (2) approaches, (3) applications, (4) tools and (5) future directions. First, we introduce the motivations, theories and insights behind CL. Second, we advocate novel, high-quality approaches, as well as innovative solutions to the challenging problems in CL. Then we present the applications of CL in various scenarios, followed by some relevant tools. In the end, we discuss open questions and future directions of this field. We believe this topic is at the core of the scope of AAAI and is attractive to the audience interested in machine learning from both academia and industry.

Graph Neural Networks (GNNs) have proven highly effective in graph-related tasks, including Traffic Modeling, Learning Physical Simulations, Protein Modeling, and Large-scale Recommender Systems. The focus has shifted towards models that deliver accurate results and provide understandable and actionable insights into their predictions. Counterfactual explanations have emerged as crucial tools to meet these challenges and empower users. For the reasons mentioned above, in this tutorial, we furnish the essential theoretical underpinnings for generating counterfactual explanations for GNNs, arming the audience with the resources to gain a deeper understanding of the outcomes produced by their systems and enabling users to interpret predictions effectively. First, we present an insight into GNNs, their underlying message-passing architecture, and the challenges of providing post-hoc explanations for their predictions across different domains. Then, we provide a formal definition of Graph Counterfactual Explainability (GCE) and its potential to provide recourse to users. Furthermore, we propose a taxonomy of existing GCE methods and give insights into every approach’s main idea, advantages, and limitations. Finally, we introduce the most frequently used benchmarking datasets, evaluation metrics, and protocols to analyze the future challenges in the field.

In the introduction (20 mins) part, we delve into the world of Graph Neural Networks (GNNs) and the growing demand for models that deliver accurate results while providing understandable and actionable insights. We start by discussing the widespread use of GNNs in diverse domains, highlighting their underlying message-passing architecture.

In the second part (30 mins), we emphasize the importance of Explainable AI (XAI) in graph-based models. Black-box models, while powerful, pose challenges in critical scenarios where interpretability is vital. We explore factual explanations, shedding light on why specific predictions are made. We also briefly revisit key methods in this category, including GNNExplainer and GraphLIME.

The core of this tutorial (55 mins) focuses on counterfactual explanations in graph-based models. We define Graph Counterfactual Explanation (GCE) and its significance, followed by a taxonomy of GCE methods. These methods include instance-level explainers (search-based, heuristic-based, learning-based) and model-level explainers. We discuss the advantages and limitations of each approach, ensuring a clear understanding of the options. Benchmarking datasets and evaluation metrics are also addressed, enabling participants to assess the performance of counterfactual explainers effectively.

Audience and Scope:

This tutorial strikes a balance between comprehensiveness and accessibility for participants with varying technical backgrounds. For those seeking a deeper dive into the technical aspects, an additional lab session is available. The tutorial caters to practitioners in academia and industry interested in applying machine learning to analyze graph data. While familiarity with basic machine learning concepts is beneficial, the content is designed to be accessible to a broad audience.

For more information and updates, visit:

https://aiimlab.org/events/AAAI_2024_Graphs_Counterfactual_Explainability_A_Comprehensive_Landscape.html

For those who want to master the field from the technical point of view, take a look at the intertwined laboratory:https://aiimlab.org/events/AAAI_2024_Digging_into_the_Landscape_of_Graphs_Counterfactual_Explainability.html

This tutorial offers an in-depth exploration into enhancing Natural Language Processing (NLP) for low-resource languages (LRLs), addressing the notable gap in current language models like ChatGPT. Focusing on languages such as Swahili, it presents strategies for data collection and model alignment to LRLs, crucial for a linguistically inclusive AI future.
Participants will gain insights into the current NLP landscape, understanding the limitations of SOTA models in current LRLs. The tutorial introduces methods to overcome these challenges, emphasizing the creation of high-quality datasets beyond traditional machine translation capabilities. This approach is vital for effectively training and aligning Large Language Models (LLMs) with LRLs.
The core of the tutorial is dedicated to demonstrating how to collect and utilize crowd-sourced data for fine-tuning LLMs for diverse linguistic needs. Attendees will learn practical guidelines and innovative techniques for aligning these models with LRLs, a crucial step in making AI technologies accessible inunderrepresented regions.
Designed for researchers and practitioners with a foundation in deep learning, this tutorial aims to support trendsfor a more inclusive and diverse AI ecosystem. Attendees will leave with a thorough understanding of the unique challenges and solutions in applying NLP technologies to LRLs, equipped to contribute to a more inclusive AI landscape.

Outline

A major drawback of deep reinforcement learning (RL) is its poor data efficiency. In this tutorial, we present meta-RL as an approach to create sample-efficient and general-purpose RL algorithms, via learning the RL algorithm itself. Meta-RL aims to learn a policy that is capable of adapting to any new task from a distribution over tasks with only limited data. We present the meta-RL problem statement, along with an overview of methods, applications, and open problems on the path to making meta-RL part of the standard toolbox for a deep RL practitioner.

Goal

The audience will come away from this tutorial with an understanding of meta-reinforcement learning. Reinforcement learning is notoriously sample-inefficient. The goal of this tutorial is to enable researchers to use meta-RL to unblock reinforcement learning from this inefficiency. Ultimately, meta-RL promises to deliver agents capable of fast and general adaptation. The tutorial should enable machine learning practitioners to get up-to-speed on meta-RL, as well as direct new research in meta-RL toward valuable and open problems.

Prerequisite Knowledge

The tutorial should be accessible to anyone with some knowledge of RL. However, minimal background is assumed. The goal is to both get people up to speed on training meta-RL agents and to point out the limitations of current systems, which will be of use to current practitioners as well.

Content

In this tutorial, we introduce the field of meta-RL following our meta-RL survey. First, we motivate reinforcement learning and meta-reinforcement learning, using supervised meta-learning for comparison. Then meta-RL research is divided into categories considering the attributes of the problem setting. One key theme that differentiates meta-RL from other meta-learning is the need for efficient exploration. The theme of exploration is covered in detail. Next, key results for exploration and generalization are presented. We discuss application domains — including robotics, education, and multi-agent RL — to make meta-RL more concrete. Next, we shift our focus to meta-learning over long horizons. We survey related methods and demonstrate the degree of transfer that can be expected. For example, this includes training on simple low-dimensional grid-world environments and then transferring the learned RL algorithm to 8-bit Atari games for evaluation. Additionally, we note open problems — including optimization challenges, benchmark standardization, and offline data collection — to catalyze further research in the field. Finally, we conclude by discussing the aims of meta-RL in the near and long term.

Deterministic search and planning typically operate on graphs where each edge has a given cost, aiming to find a path of minimal cost. Heuristic search has developed tools for solving this path-finding problem, including the A* algorithm. However, the objective of search and planning is often much more complex in real life, for example, because one needs to trade-off between different costs. Multi-objective search and planning operate on graphs where each edge has two or more given costs, aiming to find paths that trade-off between the different costs in the form of a Pareto frontier of paths.

For example, transporting hazardous material requires trading off different costs for each street, such as its length and the number of residents that would be exposed to the hazardous material in case of an accident. Other examples include route planning for vehicles and robots (which involves trading off energy consumption and travel time), planning power transmission lines (which involves trading off power-generation cost and power loss), and scheduling satellites and routing packets in computer networks (which involves trading off profit and fairness).

This tutorial will provide an overview of the multi-objective search problem and summarize recent progress in this fast-moving research area. We will cover theoretical foundations, practical algorithms, and challenges that commonly arise in practice in a way that is accessible to all AI researchers and students. Our target audience is anyone interested in search and planning who wants to learn about this fascinating emerging field.

More information on the tutorial and a detailed schedule can be found at https://sites.google.com/usc.edu/aaai24-mos-tutorial/home.

Half Day Labs

This tutorial offers a comprehensive exploration of Fully hom*omorphic Encryption (FHE) and its practical application to privacy-preserving machine learning (PPML). Aimed at data scientists, ML engineers, and students interested in FHE, the tutorial equips participants with a solid understanding of FHE concepts and hands-on experience using the OpenFHE Python library.

The tutorial covers essential FHE concepts related to ML such as noise accumulation, noise threshold levels, and noise refreshing methods. Participants will be introduced to various FHE flavors that enable ML, such as the lookup-table TFHE and Cheon-Kim-Kim-Song (CKKS) methods. This tutorial focuses on building applications with the CKKS scheme, paying special attention to various cryptographic parameters associated with it, contextualized in terms of how the parameters affect ML applications.

The second section focuses on the OpenFHE library, covering its design principles and motivation. Participants gain insights into ciphertext representation tailored for efficient batch processing in a SIMD (vectorized) fashion. Practical examples demonstrate cryptographic key generation, ciphertext encryption, decryption, and mathematical operations, emphasizing applications in machine learning. The section also provides a glimpse into the library’s development roadmap, including ports to Python and NodeJS, and collaboration with the Google Transpiler team.

The hands-on segment guides participants through training an encrypted logistic regression model using OpenFHE. We begin with a naive implementation of encrypted logistic regression training, gradually introducing optimizations, such as efficient data packing and Nesterov-accelerated gradient descent. As participants are guided through the process, we hope to build intuition and insights that will allow the attendees to learn best practices for building FHE-enabled ML applications and optimize their own programs.

The next section demonstrates how end users can transform pre-trained models into encrypted models for private inference using OpenFHE. Additionally, we benchmark the CKKS approximate number method against the lookup-table TFHE method to compare their inference performance.

The concluding section presents new ML challenges that will be posted as part of the FHERMA project (https://fherma.io/challenges). The participants will be encouraged to participate in the challenges to solidify their skills in FHE-enabled ML acquired as part of this tutorial. A brief discussion of hybrid PPML methods, where FHE is used in tandem with other privacy-enhancing technologies (PETs), such as secure multiparty computation and federated learning, is also included.

Prerequisite knowledge includes proficiency in Python programming, a grasp of logistic regression, familiarity with NumPy and Single Instruction, Multiple Data (SIMD), and a basic understanding of Support Vector Machines.

Website:

https://openfheorg.github.io/aaai-2024-lab-materials/

RDDL (pronounced as “riddle”) stands for the Relational Dynamic Influence Diagram Language. It is the domain modeling language utilized in the International Conference on Automated Planning and Scheduling (ICAPS) in the years 2011, 2014, 2018, and most recently in 2023 for the Probabilistic Planning and Reinforcement Learning track of the International Planning Competitions. RDDL was designed to efficiently represent real-world stochastic planning problems, specifically Markov Decision Processes (MDPs), with a focus on factored MDPs characterized by highly structured transition and reward functions. This tutorial aims to provide basic understanding of RDDL, including recent language enhancements and features, through a practical example. We will introduce a problem based on a real-world scenario and incrementally build up its representation in RDDL, starting from the raw mathematical equations up to a full RDDL description. Furthermore, we will introduce “pyRDDLGym,” a new Python framework for the generation of Gym environments from RDDL descriptions. This facilitates interaction with Reinforcement Learning (RL) agents via the standard Gym interface, as well as enables planning agents to work with the model. In a series of exercises, we will explore the capabilities of pyRDDLGym, which includes generation of the Dynamic Bayesian Networks (DBN) and eXtended Algebraic Decision Diagrams (XADD)-based conditional probability functions, as well as offering both generic and custom visualization options. We will also generate a functional environment for the example problem. To close the loop from a mathematical representation to a fully operational policy, we will utilize the built-in model-based anytime backpropagation planner, known as “JaxPlan.” This will enable us to obtain solutions for the example problem, effectively closing the gap between theoretical description and a practical working policy.

Recent years have witnessed an explosion in the general-purpose capabilities of AI systems, presenting unique challenges for their evaluation. Estimating capabilities, as opposed to performance, is essential for general-purpose, as opposed to task-specific, systems.
Understanding the capabilities of AI systems is crucial for anticipating their suitability in situations and occupations requiring particular cognitive skill levels to meet the expected demands. Techniques and methodologies from the cognitive sciences are more appropriate than task-oriented benchmarks for this evaluation of capability, however they require a common language and toolkit to facilitate cross-disciplinary collaboration. In this lab, we present one such approach, the Measurement Layouts framework, which leverages large, Hierarchical Bayesian Networks to infer the capabilities of AI systems. We aim to demonstrate the powerful evaluation inferences possible using this framework for various AI systems (RL agents, language models, etc.), with the hope of building a diverse community of interdisciplinary researchers committed to improving AI evaluation.
The lab comprises multiple sessions introducing participants to the Measurement Layout framework and capability-oriented AI evaluation. We will start with discussion of the limitations of current AI evaluation practices and motivate a capability-oriented evaluation, drawing on robust experimental practices from the cognitive sciences. Through hands-on practical experience, attendees will acquire the skills needed to implement the measurement layouts framework in their research and projects, adapting the material (examples, code, benchmarks, etc.) presented in this lab. Participants will actively engage in building measurement layouts in two scenarios: (i) assessing the capability of an agent in a navigation task in a 3D environment and (ii) evaluating the capabilities of large language models relative to those that might be required were such systems to take on human occupations. Finally, participants will learn to reuse and extend existing benchmarks with demand annotations to infer capabilities efficiently, as well as create new benchmarks with a grid of demands that allow the inference process to triangulate more efficiently, achieving high levels of validity.
Our target audience includes all researchers interested in improving AI evaluation. By attending this lab, participants will gain a comprehensive understanding of capability-oriented evaluation, the measurement layouts framework, and its practical applications. Specifically, they will acquire insights into how capability-oriented evaluation can enhance AI assessment beyond traditional benchmarking. Participants will learn to effectively apply the measurement layout framework, enabling them to estimate AI capabilities, and they will also gain an understanding of the benefits and challenges associated with adopting this approach.

Decision-making systems based on AI and machine learning have been used throughout a wide range of real-world scenarios, including healthcare, law enforcement, education, and finance. It is no longer far-fetched to envision a future where autonomous systems will drive entire business decisions and, more broadly, support large-scale decision-making infrastructure to solve society’s most challenging problems. Issues of unfairness and discrimination are pervasive when decisions are being made by humans, and remain (or are potentially amplified) when decisions are made using machines with little transparency, accountability, and fairness. In this tutorial, we describe the framework of causal fairness analysis with the intent of filling in this gap, i.e., understanding, modeling, and possibly solving issues of fairness in decision-making settings.
The main insight of our approach will be to link the quantification of the disparities present in the observed data with the underlying, often unobserved, collection of causal mechanisms that generate the disparity in the first place, a challenge we call the Fundamental Problem of Causal Fairness Analysis (FPCFA). In order to solve the FPCFA, we study the problem of decomposing variations and empirical measures of fairness that attribute such variations to structural mechanisms and different units of the population.
Our effort culminates in the Fairness Map, the first systematic attempt to organize and explain the relationship between various criteria found in the literature. Finally, we discuss which causal assumptions are minimally needed for performing causal fairness analysis and propose practical solutions for three key fairness tasks (in increasing order of complexity): (i) bias detection, in which disparities are analyzed and quantified; (ii) fair prediction, in which the task is to create predictions satisfying a desired notions of fairness; (iii) fair decision-making, in which the task is to implement a fair policy and achieve lower disparities over time.

Prerequisite knowledge: Attendees are expected to be familiar with standard notions in probabilistic reasoning. Familiarity with causal inference and the framework of structural causal models is desirable, but not necessary since the tutorial is self-contained, covering the key concepts of causal inference, and also introducing causal fairness from first principles.
For the practical lab component, basic familiarity with running R-code will be required, although all the code will be provided and explained during the lab. The tutorial will be entirely self-contained.

Quarter Day Labs

The rapid development of large ML models has emphasized the need for trustworthiness in the field of AI. The societal impacts of these models are enormous, and their implications require careful ethical assessment. Various regulatory bodies like the European Commission’s High-Level Expert Group on AI have stressed the need to build systems that are transparent, ensure data privacy, and can be auditable. This lab introduces artificial intelligence and machine learning metadata tracking and its importance in ensuring the trustworthiness of ML models with a Python tracking library called Common Metadata Framework (CMF).

This tutorial aims to cover the following areas:

1. Why metadata tracking is necessary for trustworthy AI.
2. Responsibilities and obligations from practitioners as outlined in various AI acts.
3. Landscape of metadata tracking tools.
4. Introduction to Common Metadata Framework.
5. Hands-on session on Common Metadata Framework to enable trustworthy AI.

The tutorial focuses on creating reproducible AI pipelines with end-to-end metadata, lineage, and provenance. It introduces integrated versioning of code, artifacts and hyper parameters associated with AI pipelines. The tutorial provides methods to share metadata between different teams or individuals in collaborative development. It also provides details about the various regulatory obligations outlined by various governance bodies and how metadata tracking enables compliance. It will also provide a high-level overview of other metadata tracking tools in the market and provide a hands-on session on CMF. Tutorial provides details on how to track metadata in complex scientific workflows with examples from model training in Fusion Data Platform and Computational steering of experiments to discover specific features in nano material using Electron Microscopy. CMF plays a central role in Fusion Data Platform to support the design and safe operation of a fusion pilot plant. Fusion Machine Learning Data Science Platform (FDP) development was selected for funding by Department of Energy. More details about CMF can be found here https://github.com/HewlettPackard/cmf.

Prerequisite knowledge:

Basic AI and ML awareness. Working knowledge of Python. AI practitioners and researchers with different backgrounds can benefit from the session. It can help early researchers, students and seasoned practitioners to understand the regulatory landscape, awareness of the new tools and frameworks that can help them to produce reproducible experiments.

In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) have emerged as potent tools with an impressive aptitude for understanding and generating human-like text. Their relevance in the domain of planning is particularly noteworthy, given the similarities between planning tasks and programming code-related tasks, a forte of LLMs. Planning, akin to scripting in the Lisp programming language using Planning Domain Definition Language (PDDL), presents a fertile ground to explore the capabilities of LLMs in devising effective and efficient plans. This lab seeks to delve deep into the nuances of utilizing LLMs for planning, offering participants a comprehensive understanding of various techniques integral to the functioning of these models. Participants will be introduced to supervised fine-tuning and a range of prompting techniques, fostering a critical analysis of which approaches tend to enhance planning capabilities significantly. At the heart of this lab is a hands-on session where participants can work closely with “Plansformer”, our proprietary fine-tuned model developed explicitly for planning tasks. This session aims to provide a comparative analysis of the current state-of-the-art LLMs, including GPT-4, GPT-3.5, BARD, and Llama2, offering insights into their respective strengths and weaknesses in planning. We will also briefly explain and show how neuro-symbolic approaches can complement the incorrect generations from LLMs.

Unveiling the inner workings behind ML decisions, counterfactual explanations offer alternative scenarios for diverse outcomes. This powerful tool empowers end users with choices, making counterfactual explainability a game-changer, especially in the dynamic realm of graph-based ML, where the heterogeneous nature of the data makes it difficult to understand the underlying relationships between the input and the outcome.

In the first part of this lab (30 minutes), we will introduce the challenges of developing and evaluating GCE methods, including a lack of standardization in metrics and oracles, as well as benchmarking studies. Then, we will present the GRETEL framework for developing and evaluating GCE methods. Furthermore, we will provide hands-on examples of how to build the explanation pipeline using GRETEL.

In the second part (30 minutes), we will present how the different categories of explainers can be implemented into the framework, including search-based methods, heuristic-based ones, learning-based ones, and global-level explainers. Moreover, we will analyze an empirical comparison of some of these methods in different datasets.

In the third part of the lab (30 minutes), we will focus on a hands-on experience demonstrating how to extend the framework to fit the needs of the users. Industry practitioners will learn how to use the framework on their datasets and to explain their own ML models, while attendants interested in XAI will learn how to develop and evaluate their explainers using the ready-to-use datasets, oracle, and evaluation metrics provided in GRETEL. Furthermore, we will show how to use the data analysis capabilities of the framework to get more insights into the explanations.

In the final section of the lab (15 minutes), we will discuss the challenges of providing Graph Counterfactual Explanation and possible ways to tackle them. This part will allow the attendants to participate in the discussion actively.

Audience and Scope:

The lab is aimed at practitioners in academia and industry interested in applying explainable machine learning techniques to analyze graph data. In particular, it is focused on the use of the GRETEL framework. Participants with a background in data mining will gain an understanding of the information provided by explanation methods to end users and how to integrate their datasets into an explanation pipeline. Those with machine learning expertise will delve deeper into state-of-the-art counterfactual explainers, critically analyzing their strengths and weaknesses. They will also learn to integrate their oracles and develop new explanation methods using the GRETEL framework. The lab will be conducted with the help of the SoBigData.it RI. Look at the website for the latest updates on the requirements.

For more information and updates, visit: https://aiimlab.org/events/AAAI_2024_Digging_into_the_Landscape_of_Graphs_Counterfactual_Explainability.html

For those interested also in the theoretical overview of Counterfactual Explanations on Graphs, there is a dedicated tutorial session:

https://aiimlab.org/events/AAAI_2024_Graphs_Counterfactual_Explainability_A_Comprehensive_Landscape.html