Experience

Audio-driven talking avatars must be judged not only by how accurately the lips follow the speech, but also by whether the face fully expresses the intended emotion. This project develops audio-driven talking-avatar generation models and studies how to evaluate them reliably and quantitatively. We examine whether multimodal LLMs can serve as judges of the emotion expressed by generated 3D talking heads and investigate how they perceive different visual representations of rendered videos.

Federated learning is expected to become a foundational technology for embodied AI and robotics, where multiple agents need to learn collaboratively. In such multi-agent systems, each agent collects its own data in a distributed manner, and the agents must learn together from the data that is reliable. This project is aimed at developing a collaborative learning framework for multi-agent systems, focused on federated learning within the context of data-heterogeneity among multiple agents. We aim to develop both general and personalized federated learning algorithms that surpass existing state-of-the-art solutions.

XAI-based Infectious Disease Spread Modeling Jul 2021 – Dec 2021

The global pandemic of COVID-19, first reported in China in December 2019, is causing many people to suffer. This pandemic caused each government to make mistakes of increasing the damage and scale without responding properly. The existing statistical or SIR spread model is still used by government quarantine officials to measure spread rate of infectious diseases. However, there is a lack of consideration for the actual impact of each policy on the spread of infectious diseases between regions or countries. In addition, various factors such as people’s mobility change or distancing policies are difficult to be considered with SIR models. In order to establish and implement prevention policies early, it is necessary to predict the spread of infectious diseases and understand the influence of policies by considering various complex factors. In this study, we develop an early prediction model for the spread of infectious diseases, based on explainable artificial intelligence (XAI) to provide a basis for responding properly in future crisis situations.

Outputs: Technical report, Code

Detection and Prediction Model for Infectious Disease Feb 2022 – Dec 2022

The global pandemic of COVID-19 that has continued from 2019 to the present indicates the importance of early diagnosis of infectious diseases. If we can detect in advance that a disease has a high risk of spreading significantly, it will not only be able to specify information that needs to be searched, but also prevent the spread of infectious diseases by appropriate early responses. In this project, we develop an infectious disease monitoring/surveillance model, so that important information is analyzed to quickly grasp the risk in the early stages of infectious diseases. Also, by integrating the monitoring model and the spreading model, we enable complex analysis of infectious diseases.

Outputs: Code

Data-efficient Unsupervised Representation Learning Mar 2020 – Dec 2020

We are often unavailable to obtain enough labeled data for our target task (e.g., image classification) in most real-world cases. Some recent works have addressed the data shortage by proposing un-/semi-/self-supervised learning methods that utilize unlabeled data to improve the performance in target task without or with only a few amounts of labeled data. In this project, we extend those methods to build a data and training efficient algorithm to cope with various real-world constraints.

Outputs: Code

Semantic Segmentation Error Detection Feb 2021 – Sep 2021

It is crucial for the model not only to accurately predict, but also to predict how certain it is. If we can find areas where the model's predictions are uncertain, it can greatly reduce the labor force because a person can intervene and correct only those parts. In the semiconductor industry, AI models (semantic segmentation models) segment wafer photos for the quality control of semiconductor wafers, but the amount is too large and ultra-high resolution, making it difficult for humans to inspect them one by one. After the segmentation model makes a prediction, if only the unclear parts are given to a person, he/she can correct those few samples. Therefore, in this work, we intend to create a model that filters out uncertain data by introducing a learning methodology that predicts how certain the prediction of the image segmentation model is.

Outputs: Code

Semantic segmentation models used in safety-critical applications such as autonomous driving need to account for how uncertain their predictions are. In this project, we study uncertainty modeling and calibration for semantic segmentation and develop an uncertainty-aware semi-supervised model, using video data recorded from real vehicles.