Welcome to EchoSight, an innovative deep learning project focused on creating a joint embedding space for audio and video modalities. Our primary goal is to explore and develop a model that can effectively learn from both audio and video data simultaneously, utilizing the inherent relationships between these two modalities for enhanced representation and understanding. Inspired by the success of the CLIP model for image and text joint multimodal contrastive learning, EchoSight extends this approach to the audio-visual domain. Our model is designed to capture the interdependencies between audio and video features, enabling a variety of applications such as multimodal search, content recommendation, and content generation. EchoSight is built upon a cutting-edge deep learning architecture, leveraging state-of-the-art techniques in multimodal learning and contrastive optimization. The idea is to train the model on a large-scale dataset consisting of diverse audio-visual content, effectively learning the intricate relationships between sounds and visuals in a wide range of contexts.
Right now, the project is a work in progress. The first version of the model (EchoSight v0.1) was trained a small dataset (VEGAS that now appears to be down...) which consists on a small set of classes (snoring, baby crying, water flowing, ...) and 21k videos. As mentioned above, the idea is to train the model on a bigger and more general dataset like YouTube-8M.
We won't provide any metrics yet as the model is yet to be defined. But we share some preliminary results. Note that the lengths of the video and audio are not seen by the model as they are internally cropped, we've included the full audio and video for aesthetic purposes.
The video represents a (scary) barking dog.
The model succesfully choses the correct audio (first one) as the closest to the video.
The video represents flowing water in a pool with some other external noise.
The model succesfully choses the correct audio (first one) as the closest to the video.