Carnegie Mellon University
Multimodal Activity Recognition and Prediction for Assisting Older Adults with MCI to Prepare Meals
This research project seeks to enhance the independence and efficiency of older adults with mild cognitive impairments during meal preparation activities, a key iADL faced by many individuals with MCI. This project will first capitalize on existing AI-CARING datasets of meal preparation to understand the dynamics between caregivers and older adults, drawing insights on task progression, patterns and behaviors, and techniques in which caregivers provide assistance during meal preparation. We will then develop an AI system that fuses first-person camera observations with wearable smartwatch IMU data to recognize distinct meal preparation activities and then, building on prior AI-CARING research by Daphne Chen, predict future activities based on historical observations.
The ultimate aim of this project is to predict fine-grained sequential activities of meal preparation, and determine optimal moments to offer prompts or support, while preserving the users’ autonomy. In contrast to high-level task progression studied in prior research, this project will highlight an individual’s detailed interactions with tools, utensils, foods, and their caregivers throughout meal preparation, such as chopping tomatoes with a knife, receiving a potato from a caregiver/partner, peeling potatoes with a peeler, pouring sauce from a can by hand, mixing with a large spoon, and more. In addition to small human studies at CMU to evaluate prototypes of our models, we will capture meal preparation activity data with older adults in independent living, personal care, and nursing homes residents, leveraging existing collaborations and partnerships with communities in Pittsburgh that already work with us on other projects. Moreover, we anticipate collaborating and sharing the rich data gathered from diverse living settings with other researchers in the institute as a resource to support further advancements towards assistive agents for a key challenging task—meal preparation—faced by many individuals with MCI.