Blind Man's Buff - George Morland (1788)
Humans are provided with a remarkable cognitive framework that allows them to process complex perceptual information and create a rich representation of their external reality. These perceptual representations play a fundamental role in the planning and execution of tasks, even in challenging perceptual conditions such as the absence of modality-specific information (e.g. navigating a dark room) or if a given sensor is malfunctioning (e.g. visual impairments).
We aim at endowing agents with mechanisms to learn representations from multimodal sensory data and to employ them in the execution of tasks.
How can artificial agents learn multimodal representations of their environment and leverage such representations to act robustly in challenging perceptual conditions?
A multimodal representation learning model based on contrastive learning that allows the execution of downstream tasks with missing modality information.
In International Conference on Machine Learning. PMLR, 2022. (pp. 17782-17800). [pdf]
A multimodal generative model for the perception of RL agents that allows the execution of control tasks with missing observations.
In Proceedings of the 21th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1301-1309. 2022. [pdf]
Multimodal Transfer RL
An architecture that allows a RL agent to reuse a policy trained over a given sensory modality (e.g. image) on a different sensory modality (e.g sound)
In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1260-1268. 2020. [pdf]
Doctoral Consortium of the 19th International Conference on Autonomous Agents and MultiAgent Systems (2020) - 9 min