Carnegie Mellon University
I am interested in providing personalized guidance and assistance, especially to older adults with cognitive issues. We are looking at learning people's preferences through observation and interaction with them, and then using those preferences to tailor subsequent interactions. In particular, we are exploring how AI agents can learn how people typically prepare meals, watch for when they seem to be stuck at some step in the process, and then provide gentle feedback to get them back on track. We are similarly exploring how an AI agent can learn the type of feedback that people find most useful while doing exercises, tailoring the style (e.g., encouraging vs. firm) to fit their preferences, level of fatigue, etc. Along those lines, we are interested in AI agents that can both generate and understand non-verbal feedback, such as emotion, posture, and gesture. For instance, we need to be able to tell when a person is having difficulties determining what to do next during meal preparation - what expressions are telling, what about their gaze and posture? Similarly, for exercising, how to tell from people's faces and posture if they are in pain, disengaged, pushing themselves?
Other research of interest includes combining the vast knowledge embodied in Large Language Models (LLMs) with the functionality provided by cognitive architectures, which model how some people believe the mind works. We have demonstrated how LLMs can bootstrap the knowledge needed by a cognitive architecture, which can then learn what knowledge is actually useful in performing task and what can be ignored in given situations. In addition, human interaction can help the agent refine it's knowledge, either correcting things that the LLMs got wrong or providing modifications that indicate human preferences. Finally, we are interested in modeling and reasoning about social norms. Norms are extremely important in human-human interaction, and providing AI agents with the ability to reason about norms, especially tradeoffs between them (e.g., do you violate someone's privacy in order to prevent them from harming themselves) is particularly germane when dealing with older adults who have cognitive difficulties.