Artificial intelligence has significantly advanced in developing systems that can interpret and respond to multimodal data. At the forefront of this innovation is Lumos, a groundbreaking multimodal question-answering system designed by researchers at Meta Reality Labs. Unlike traditional systems, Lumos distinguishes itself by its exceptional ability to extract and understand text from images, enhancing the…

The emergence of Multimodality Large Language Models (MLLMs), such as GPT-4 and Gemini, has sparked significant interest in combining language understanding with various modalities like vision. This fusion offers potential for diverse applications, from embodied intelligence to GUI agents. Despite the rapid development of open-source MLLMs like BLIP and LLaMA-Adapter, their performance could be improved…

The capacity of infographics to strategically arrange and use visual signals to clarify complicated concepts has made them essential for efficient communication. Infographics include various visual elements such as charts, diagrams, illustrations, maps, tables, and document layouts. This has been a long-standing technique that makes the material easier to understand. User interfaces (UIs) on desktop…

Big Vision Language Models (VLMs) trained to comprehend vision have shown viability in broad scenarios like visual question answering, visual grounding, and optical character recognition, capitalizing on the strength of Large Language Models (LLMs) in general knowledge of the world. Humans mark or process the provided photos for convenience and rigor to address the intricate…

In recent years, LMMs have rapidly expanded, leveraging CLIP as a foundational vision encoder for robust visual representations and LLMs as versatile tools for reasoning across various modalities. However, while LLMs have grown to over 100 billion parameters, the vision models they rely on need to be bigger, hindering their potential. Scaling up contrastive language-image…

Personalized image generation is the process of generating images of certain personal objects in different user-specified contexts. For example, one may want to visualize the different ways their pet dog would look in different scenarios. Apart from personal experiences, this method also has use cases in personalized storytelling, interactive designs, etc. Although current text-to-image generation…

View synthesis, integral to computer vision and graphics, enables scene re-rendering from diverse perspectives akin to human vision. It aids in tasks like object manipulation and navigation while fostering creativity. Early neural 3D representation learning primarily optimized 3D data directly, aiming to enhance view synthesis capabilities for broader applications in these fields. However, all these…

The advancement of AI has led to remarkable strides in understanding and generating content that bridges the gap between text and imagery. A particularly challenging aspect of this interdisciplinary field involves seamlessly integrating visual content with textual narratives to create cohesive and meaningful multi-modal outputs. This challenge is compounded by the need for systems that…

Current challenges faced by large vision-language models (VLMs) include limitations in the capabilities of individual visual components and issues arising from excessively long visual tokens. These challenges pose constraints on the model’s ability to accurately interpret complex visual information and lengthy contextual details. Recognizing the importance of overcoming these hurdles for improved performance and versatility,…

Deep convolutional neural networks (DCNNs) have been a game-changer for several computer vision tasks. These include object identification, object recognition, image segmentation, and edge detection. The ever-growing size and power consumption of DNNs have been key to enabling much of this advancement. Embedded, wearable, and Internet of Things (IoT) devices, which have restricted computing resources…