Our recent research focuses on trustworthy AI, aiming to create AI systems that are explainable, generalizable, and fair. Below are several current topics:
- Explainable AI: our research highlights explainability that encodes and presents the decision-making process of deep learning models (e.g., neural module networks, reasoning-aware attention). We also develop large-scale explanation data and models.
- Generalizable AI: our research further enables AI to perceive and reason with commonsense or domain-specific knowledge as humans do. Our recent knowledge incorporation and augmentation methods achieve state-of-the-art accuracy and generalizability. We also develop compositional methods that enhance AI's robustness and adaptability.
- Fair AI: Our work on fairness focuses on developing data and methodologies to detect and mitigate biases in AI systems, ensuring equitable outcomes.

We are interested in AI systems that perceive, learn, and reason to perform real-world tasks. Below are several current topics:
- Attention: Our research shows that attention is an essential component in AI models for them to perform a variety of tasks (e.g., social relationship recognition, video storytelling, and LLM-based vision-language reasoning).
- Reasoning: We also look into the information that is being attended overtime and its coupling with reasoning - the two intertwined mechanisms in decision making. We develop frameworks to study the two for both humans and models, and innovate AI models that are able to iteratively attend and reason to perform AI tasks (e.g., AiR, IQVA).
- Problem Solving: our VisualHow dataset and models leverage attention and reasoning mechanisms to understand and solve real-life problem solving.

We study saliency and make a number of data and model contributions to unaddressed issues in saliency research (see here for details). SALICON is an ongoing effort in understanding and predicting attention in complex natural scenes with big data paradigm and data-driven approaches. So far we have:
- Innovated a new experimental method for gaze collection that enables: crowdsourcing gaze data (for both computational and clinical studies), and studies with groups not good with eye trackers (e.g., kids, chimps).
- Collected the largest-to-date gaze data for training and benchmarking saliency models as well as encouraging methods that leverage multiple annotation modalities from MS COCO.
- Developed a deep neural network based model that bridges the "semantic gap" in predicting where people look, and currently ranks top in the MIT Saliency Benchmark.
- Developed an adversarial network to anticipate future gaze in videos.
- Co-hosted the large-scale saliency challenge in LSUN workshop yearly at CVPR.

We propose a number of theoretical and application innovations in machine learning methods. We study methods to make deep neural networks more efficient, generalizable, and trustworthy. We develop representations and architectures for visual, behavioral and neural data, addressing domain-specific needs and overcoming their challenges. We are also interested in visualizing and interpreting deep neural networks.
- Trustworthiness prediction
- Enhancing congruency in machine learning
- Learning to learn from noisy labeled data
- Efficient deep neural networks
- Unsupervised representation learning

We develop neural networks and other AI technologies to understand neurodevelopmental and neuropsychiatric disorders for automatic screening and improved personalized care. For example, atypical attention and behaviors play a significant role in developing impaired skills (e.g., social skills) and reflects a number of common developmental / psychiatric disorders including ASD, ADHD, and OCD. Our study integrates several techniques including behavioral, fMRI and computational modeling to characterize the heterogeneity in these disorders and to develop clinical solutions. We are lucky to work with leading scientists and clinicians including Ralph Adolphs, Jed Elison, Suma Jacob, Christine Conelea, Sherry Chan, and Kelvin Lim.
- Identifying people with autism with deep learning and multi-modal distillation.
- Classifying ages of toddlers based on machine learning and eye-tracking
- Atypical attention in autism quantified through model-based eye tracking
- Revealing the world of autism through the lens of a camera

We are interested in areas bridging artificial intelligence and human functions. We have developed a neural decoder to infer human motor intention based on peripheral nerve neural recordings and demonstrated the first 15 degree of freedom motor decoding with amputee patients. It has also shown an example of true bi-directional human-machine interface with mutual learning and adaptation: the human learns over time to better express their intent to mind-control prosthetic hand movement, and the machine learns from the choices and refined the model. We collaborate with talented engineers, scientists, and clinicians on this exciting topic: Zhi Yang, Edward Keefer, and Jonathan Cheng.
- Artificial intelligence enables real-time and intuitive control of prostheses via nerve interface
- A bioelectric neural interface towards intuitive prosthetic control for amputees
- Human motor decoding from neural signals: a review

The critical need to transform the US industry is driven by severe supply chain disruptions, complex technology demands, labor shortages, and declining interest among the younger workforce. To overcome these obstacles, we develop trustworthy AI systems that automate manufacturing processes, significantly enhancing their efficiency through improved monitoring and control. Our approach involves equipment sensorization, dataset standardization, and integration of robust and trustworthy AI models into advanced process control systems. These efforts aim to optimize process yield and quality, thereby advancing manufacturing capabilities in broad areas such as semiconductor and other industries. We collaborate with Seagate Technology to advance the use of machine learning in the manufacturing.
- IGAIM: Interpretable and Generalizable AI for Smart Manufacturing