Humans experience and interact with the world through sensory information. This could mean learning correspondences between the sound of bird chirps and the word “robin,” or being able to visually identify and follow your friend “Sarah” in a crowd of people. In many cases, people are able to draw on their latent “multimodal” knowledge — information across multiple data types, from audio to video to text or spoken word — to perform these tasks without many, or sometimes even any, examples.
Multimodal foundation models of artificial intelligence are increasingly gaining capabilities that mimic this. However, their apparent multimodal fluency often belies their actual understanding. An MIT-IBM Computing Research Lab team, co-led by Jim Glass, senior research scientist in the MIT Computer Science and Artificial Intelligence Laboratory, has been investigating critical failures of state-of-the-art multimodal models and developing benchmarks and data-efficient methods to overcome these performance gaps with limited supervision. He speaks about this work in the following interview.
Q: What were your goals for this project, and why would you opt to address AI models’ weaknesses over scaling to larger…